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Introduction 


The  grant  was  begun  in  2008  and  slightly  realigned  in  2009  with  a  different  research  team 
under  TATRC’s  guidance  and  formal  approval.  The  primary  goal  of  this  project  has  been  to 
design  and  develop  an  interactive  collaborative  team  training  simulator  that  persuades 
users  to  perform  a  sequence  of  cognitive  as  well  as  psychomotor  actions  in  time- 
constrained  environment.  The  events  take  place  with  a  virtual  learning  environment  which 
includes  collaborative  work  done  by  users  who  are  located  at  different  sites  integrated 
with  virtual  environments  called  Collaborative  Virtual  Environments  (CVE.)  These  provide 
immersive  virtual  environments  where  users  can  perform  various  actions  and  can  also 
communicate  and  collaborate  with  others  in  the  environment.  Training  in  the  CVE  with 
Advanced  Cardiac  Life  Support  (ACLS)  will  be  evaluated  for  both  initial  learning  as  well  as 
retention  and  degradation  of  skills  over  time.  Virtual  training  has  the  ability  to  deploy 
more  persuasive  technologies  and  thus  has  the  potential  of  having  a  greater  impact  on 
changing  behaviors  of  the  subjects.  These  hypothesized  changes  on  the  learning  and 
behaviors  of  clinical  providers  will  be  studied  in  this  project. 


Body: 

During  2011  several  transitions  occurred  in  the  research  team  on  this  research  project.  In 
June  the  Principal  Investigator  for  Arizona  State  University  Kanav  Kahol  PhD  accepted  an 
offer  of  employment  with  the  government  of  India  to  work  on  a  global  health  plan  for  all 
citizens  of  his  home  country.  Robert  Greenes,  M.D.,  PhD,  Director  of  the  Biomedical 
Informatics  Department  at  Arizona  State  University,  then  assumed  the  role  of  the  principal 
Investigator  for  Arizona  State  University,  who  is  a  subcontractee  of  Banner  Health  for  this 
research  project.  Dr.  Kahol  was  then  transitioned  to  the  role  of  Consultant  and 
Independent  Contractor  for  the  last  year  of  the  grant  because  of  both  his  overall  familiarity 
with  the  project  and  the  short  duration  until  the  grant  was  finished.  It  was  felt  this  was  in 
the  best  interests  of  finishing  within  the  allocated  timeframe,  and  all  of  these  changes  were 
done  with  the  full  approval  of  the  Telemedicine  and  Advanced  Technology  Research 
Center’s  legal  team  and  the  Grants  program  Manager. 

After  developing  the  initial  framework  for  virtual  worlds  and  developing  the  underlying 
architecture  in  the  first  two  years,  this  year  our  experiment  focused  improving  the  virtual 
experience  of  the  user  in  the  clinical  area  and  on  the  development  of  algorithms  in 
cardiopulmonary  resuscitation.  Cardiopulmonary  arrest  (more  commonly  known  as 
cardiac  arrest)  is  the  absence  of  mechanical  activity  of  heart  or,  abrupt  loss  of  functionality 
of  heart.  According  to  the  American  Heart  Association,  almost  80  percent  of  cardiac  arrests, 
which  occur  out  of  hospital,  are  witnessed  at  home  by  a  family  member  [33]. 

Approximately  6.4  percent  of  the  patients  who  have  a  cardiac  arrest  ultimately  survive 
[33].  This  shows  the  importance  of  Advanced  Cardiac  Life  Support  (ACLS)  skills,  which 
requires  a  team  to  perform  various  tasks  within  a  few  minutes  of  patients’  arrival  in 
emergency  room.  It  is  a  time-constrained,  sequential  procedure  and  complex  team  event 
that  requires  fluid  communication  and  coordination  between  the  team  members  in  order 
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to  save  a  patient’s  life.  The  ACLS  team  has  only  five  minutes  to  perform  the  sequence  of 
actions,  both  cognitive  (eg.  decision-making  such  as  which  medicine  to  give,  diagnosis  of 
treatment  scenario)  and  psychomotor  (eg.  CPR),  in  order  to  save  a  patient.  Success  is  more 
dependent  upon  the  performance  of  the  team  rather  the  individuals  during  these 
procedures.  However,  if  a  team  member  makes  a  mistake  then  this  can  undermine  the 
work  done  by  the  team  and  the  patient’s  life  may  not  be  saved. 

In  most  fields  in  which  time  is  the  most  important  factor  and  which  require  expertise  in 
both  cognitive  and  psychomotor  skills  for  better  decision-making,  novices  require  an 
expert  to  disseminate  knowledge  and  skills  to  them.  Theoretical  knowledge  can  be  learned 
in  classroom  environments  whereas  procedural  skills  and  communication  skills  require 
more  hands  on  practice  to  perfect.  This  approach  of  master-apprenticeship  (or 
apprenticeship  in  common)  model  of  education  has  been  in  existence  for  many  years, 
where  an  expert  performs  a  procedure  and  trainees  carefully  observe  the  procedures  and 
practice  them.  From  theoretical  knowledge  to  procedural  skills  to  communication  skills, 
this  model  best  fits  the  requirements.  In  the  case  of  psychomotor  and  communication  skills, 
this  is  more  important,  because  initially  most  of  the  trainees  do  have  any  concept  of  what 
they  are  going  to  do  during  their  initial  learning  phase.  However,  there  is  a  limitation  that 
at  any  point  of  time,  a  trainer  can  only  train  a  limited  number  of  trainees  [1]. 

Hamman,  Beaubien,  and  Seiler  [34]  present  the  fact  that  errors  in  health  care  are  directly 
related  to  the  failures  in  the  structure  and  function  of  the  systems.  The  authors  also 
mention  that  team  training  is  given  less  preference  than  training  an  individual,  although 
most  of  the  care  delivery  is  performed  by  teams  of  people.  As  mentioned  earlier,  ACLS  is  a 
team-based  time  critical  event,  so  in  order  to  deliver  better  care  to  the  patients,  it  is 
important  to  understand  the  significance  of  team  training  as  well  as  to  consider  more 
effective  ways  to  provide  training  for  these  teams. 

What  is  ACLS? 

ACLS  refers  to  certain  clinical  interventions  intended  to  treat  life  threatening  medical 
emergencies  such  as  cardiac  arrest  and/or  respiratory  failure  etc.  To  master  ACLS,  it 
requires  extensive  medical  knowledge,  training  and  practice.  Only  qualified  healthcare 
providers  such  as  physicians,  nurses  and  paramedics  etc.  can  provide  ACLS  as  it  requires 
several  qualified  skills  such  as  understanding  emergency  pharmacology,  managing 
patient’s  airway  and  interpreting  electrocardiograms. 

Life  threatening  situations  as  cardiac  arrests  are  announced  as  a  code  blue  situation.  Code 
Blue  is  one  of  the  common  hospital  emergency  codes  that  are  used  to  alert  several 
emergency  situations  in  hospitals  worldwide  so  that  the  staff  is  aware  and  able  to  react  to 
the  situation  as  fast  as  possible.  Code  Blue  means  that  there  is  a  patient  who  is  suffering 
from  a  severe  life  threatening  situation  and  he  requires  immediate  resuscitation  (needs 
ACLS).  There  is  a  dedicated  team  of  nurses  who  specialize  in  responding  to  these  situations. 
The  team  needs  to  be  aware  of  the  situation  quickly  and  act  accordingly  since  the  patient 
needs  immediate  attention.  The  delays  in  response  may  result  in  death  of  the  patient. 
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Once  the  code  team  has  assembled  they  have  to  follow  predefined  protocols.  They  have 
very  limited  time  to  react  and  it  is  literally  a  life  or  death  situation.  They  are  bound  to  make 
some  mistakes  under  this  stress  and  tension.  Our  hypothesis  is  that  we  can  decrease  the 
error  rate  significantly  if  we  focus  on  the  procedural  aspect  of  their  activities. 

ACLS:  current  training  approach 

Almost  all  patient-care  organizations  provide  regular  ACLS  training  to  facilitate  emergency 
care  providers  to  enhance  their  ACLS  skills.  In  a  typical  training  session,  training  team 
members  arrive  at  the  practice  room.  They  initiate  the  process  by  assigning  roles  at  first, 
then  divide  the  tasks  according  to  the  roles,  and  follow  the  tasks.  The  team’s  performance  is 
monitored  and  evaluated  by  an  evaluator  throughout  the  period.  After  the  session,  the 
evaluator  gives  a  final  score  based  on  the  team’s  performance,  and  later  s/he  debriefs  what 
happened  and  what  should  have  been  done  in  the  practice  room.  There  might  be  a  brief 
didactic  session  on  ACLS  too.  After  the  debriefing  session  (and  the  didactic  session),  the 
team  will  perform  another  test,  and  the  team  is  expected  to  perform  better  than  the 
previous  session.  The  same  evaluator  will  evaluate  the  second  session  as  well. 

The  problem(s) 

Although  the  current  training  methodology  looks  comprehensive,  there  are  various  issues 
that  are  sub-optimal.  The  cost  associated  with  overall  setup  falls  on  a  higher  range,  and  the 
time  taken  for  training  takes  about  2  to  3  hours  to  complete.  Much  of  this  time  is  due  to  the 
large  amount  of  orientation  needed  for  training.  In  the  context  of  learning,  the  training 
participants  are  not  guided  during  the  practice  session.  So,  they  have  to  recall  what  they 
had  learned  previously  in  the  didactic  session.  There  are  rarely  adequate  trainers  to 
provide  training  to  the  trainees  frequently  because  of  which  trainees  get  less  time  to 
practice  the  procedures  properly.  Apart  from  these,  the  ACLS  training  sessions  occur  only 
once  in  every  three  weeks,  which  is  not  enough  for  practice  when  we  consider  the 
criticality  of  the  ACLS  skills. 

Learning  in  virtual  worlds 

With  rapid  development  of  computer  storage,  memory,  processors,  and  high  speed 
network  infrastructure,  it  is  now  possible  to  create  a  virtual  reality  based  simulations  in  a 
networked  (distributed)  environment  that  helps  users  to  learn  team  coordination  skills. 
Computer  Supported  Cooperative  Work  (CSCW),  in  general  terms,  is  considered  to  be  a 
collaborative  work  done  by  users  who  are  located  at  different  sites.  Telemedicine,  tele¬ 
health,  teleconferencing  all  are  examples  of  CSCW.  When  CSCW  is  integrated  with  the  term 
Virtual  Reality  (VR),  the  environment  is  called  as  Collaborative  Virtual  Environments 
(CVE),  or  simply  “Virtual  Worlds",  which  provide  immersive  virtual  environment  where 
users  can  perform  various  actions,  and  can  also  communicate  and  collaborate  with  others 
in  the  environment.  CVEs  have  been  used  in  various  fields  like  gaming  [4],  online 
community  building  or  socializing  [4,  5],  educational  or  working  environments  [6,  7].  CVEs 
are  able  to  convey  the  social  dynamics  like  turn  taking,  cooperation,  appraisal, 
communication  to  users  in  a  proper  manner.  In  addition  to  that,  users  can  be  assigned 
different  roles  like  doctor,  patient,  trainer,  trainee  etc.  Current  CVEs  also  support  different 
media  required  for  communication  (text,  audio,  video),  which  are  very  important  for  group 
discussions. 
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How  virtual  worlds  can  persuade  users  to  change  their  behavior  and  attitude 

Because  of  the  features  that  virtual  worlds  provide,  they  have  potential  to  change  behavior 
and/or  attitude  at  different  situations  and  different  circumstances.  Fogg  mentions  that 
there  are  many  reasons  that  computers  can  be  better  persuaders  than  humans  [8].  Some  of 
the  important  reasons  are:  computers  are  more  persistent;  they  provide  greater 
anonymity;  they  can  offer  various  modalities;  computer  programs  can  be  re-scaled  as  per 
users’  need;  and  the  most  important  one  -  “computer  can  be  ubiquitous".  Virtual  worlds 
provide  all  these  features.  They  are  more  persistent;  they  are  able  to  hide  users’ 
information;  various  input  output  methods  can  be  integrated  with  the  virtual  worlds;  and 
can  be  modified  as  per  the  requirements.  In  presence  of  internet,  virtual  worlds  can  be 
accessed  from  any  part  of  the  world.  Hence,  we  can  say  that  CVEs  are  an  integral  part  of 
persuasive  framework  in  various  fields  like  gaming  (eg  World  of  Warcraft), 
communications  (eg.  virtual  shops:  Amazon.com,  eBay.com  etc.],  training  systems  for 
physical  exercise  (eg.  virtual  trainers:  TripleBeat,  Wii  Fit  etc).  With  these  abilities, 
computerized  virtual  reality  based  interactive  systems  have  potential  to  persuade  human 
users  in  the  field  of  education  as  well. 

Advantage  (s)  of  training  in  virtual  worlds 

The  most  important  advantage  of  use  of  computer  based  simulation  in  the  field  of 
education  is  that  it  can  motivate  students  to  learn  and  practice  in  a  safe  environment  [9]. 
Simulation  also  enables  students  to  practice  different  procedures  in  different  contexts  and 
different  situations.  Chodos  et.  al.  suggest  that  virtual  world  simulations  consume  less 
resources  and  are  capable  of  providing  safe  and  realistic  environment  to  practice  [1].  The 
added  persuasion  in  the  computer  simulation  allows  students  to  learn  what  the  causes  are 
and  the  effects  caused.  This  persuades  students  to  enhance  their  skills  on  role-playing,  and 
changing  their  attitudes  towards  different  perspectives  [8]. 

Learning  in  virtual  worlds:  what  is  required? 

Research  on  team  training  in  CVEs  is  still  at  its  infancy.  Current  applications  of  CVEs  do  not 
consider  implementation  of  time-critical  high-performance  system.  They  still  lack  the 
integration  of  cognitive  task  as  well  as  psychomotor  task  by  providing  an  interactive 
platform  to  users  to  perform  the  tasks.  There  are  many  team  based  activities  which  include 
sequence  of  actions  and  are  constrained  by  time  and  team-members  have  to  complete  their 
task  within  the  specified  time  frame  maintaining  high  performance  level.  Research  studies 
have  shown  adverse  effects  of  time  criticality  on  performance  of  users,  which  also  hinders 
the  persuasiveness  in  the  CVEs.  In  virtual  worlds,  it  is  more  likely  that  users  will  pay  less 
attention  while  performing  time-critical  procedure.  The  lack  of  verbal  interaction,  physical 
cues  (like  facial  expressions,  eye  movement,  movement  etc.),  and  psychological  cues  (like 
feelings,  humor,  preferences  etc.)  are  also  major  barriers  for  the  implementation  of 
persuasion  in  the  virtual  worlds. 

Contribution  and  hypothesis 

In  this  study,  we  attempt  to  address  the  issue  of  team  training  in  time  critical  scenario 
(ACLS  in  our  case)  and  also  the  learning  behavior  of  participants  in  different  scenarios: 
when  the  participants  are  provided  with  persuasive  elements  and  when  they  are  not.  We 
then  discuss  whether  the  participants  can  transfer  the  learned  skills  to  the  training  room  at 
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a  hospital.  Finally,  we  also  see  whether  the  participants  can  retain  the  skills  in  the  virtual 
world.  We  also  discuss  the  novel  approach  of  integration  of  haptic  device  to  the  virtual 
world  for  time  critical  activities  that  requires  psychomotor  skills.  After  the  study,  we 
predict  that: 

Hypothesis  1#:  Virtual  worlds  are  significantly  effective  in  delivering  team  training. 
Hypothesis  2#:  Participants  will  retain  the  skills  over  a  longer  period  of  time. 

Objectives 

CVEs  have  a  huge  potential  to  provide  training  to  many  users  in  a  virtual  environment 
simultaneously.  Our  primary  goal  of  this  study  is  to  design  and  develop  an  interactive 
collaborative  team  training  simulator  that  persuades  users  to  perform  a  sequence  of 
cognitive  as  well  as  psychomotor  actions  in  time-constrained  environment. 

The  study  also  focuses  on  the  following  important  issues: 

•  Evaluate  the  validity  of  virtual  worlds  in  delivering  team  training  and  retention  over 
a  long  period  of  time. 

•  Monitor  and  record  activities  (and  hence  performance)  of  users  while  performing  a 
collaborative  task. 

•  Create  an  online  result  sheet,  which  can  be  accessed  from  anywhere  to  view  own 
performance.  (The  security  feature  of  the  performance  sheet  can  be  customized: 
teams  can  view  only  their  results;  whereas  a  supervisor  can  view  all  results). 

Background 

The  project  commenced  in  October  2008.  From  a  financial  perspective,  many 
original  quotes  for  equipment  were  no  longer  valid  due  to  significant  price  increases  of  the 
equipment  since  the  original  proposal  was  submitted.  This  limited  the  ability  to  complete 
the  proposed  project  for  developing  physical  telemedicine  connections  across  the  western 
region  of  Banner.  More  importantly,  the  project  did  not  have  a  clinical  champion  as  the 
Principle  Investigator  and  that  would  have  been  a  major  roadblock  in  accomplishing  the 
goals  of  collaborative  telemedicine.  These  factors  were  recognized  within  the  first  three 
months  of  the  project,  and  at  which  stage  TATRC  was  informed  about  the  difficulties  that 
had  arisen.  Arizona  State  University  (ASU)  continued  to  develop  the  web  2.0  backbone  for 
the  project,  but  the  project  was  halted  at  that  point.  At  this  stage  TATRC  was  contacted  and 
engaged  to  better  define  a  new  project  within  the  lines  of  military  relevance  and  of 
importance  to  our  organization.  Banner  Health  presented  a  new  plan  to  TATRC  and  it  was 
approved  on  June  12,  2009.  The  actual  project  started  in  July  2009.  Since  the  project  start 
we  have  made  some  rapid  achievements  in  laying  the  foundation  of  the  virtual  world. 

To  lay  the  foundations  of  our  work,  we  will  present  the  related  work  and  then  highlight  our 
conceptual  framework. 

Related  Work 

We  sub  categorize  this  section  into  three  parts:  Team  Training,  Training  in  Virtual  Worlds, 
and  Persuasive  Technologies. 
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Team  Training 

Any  coordinated  effort,  performed  by  a  number  of  people  in  a  group  is  termed  as  team 
work.  Communication,  coordination,  cohesion  etc.  are  typical  characteristics  of  a  team.  All 
the  team  members  should  possess  these  skills  in  order  to  carry  out  assigned  task.  Team 
training  is  very  crucial  if  well  coordinated  team  work  is  required. 

Today,  almost  every  single  case  of  care  delivery  in  hospitals  or  outside  hospitals  involves  a 
team  of  healthcare  professionals  yet  it  has  been  observed  that  more  often  than  not, 
individual  training  is  given  more  importance  in  real  life  [27].  There  are  various  reasons 
behind  this  fact  such  as  it  is  often  hard  to  set  up  training  sessions  according  to  each 
individual’s  schedule,  health  care  professional  trainees  are  from  disparate  locations  etc. 
These  healthcare  training  programs  need  to  increase  training  experience  of  working  in 
interdisciplinary  teams  for  every  individual  caregiver.  Hamman  W  et  al  demonstrated  that 
identifying  and  focusing  on  team  critical  tasks  and  events  prior  to  and  during  the  training 
respectively,  actually  lead  to  significant  performance  improvement  in  teamwork  skills  [27]. 

Implicit  coordination  is  one  of  the  characteristics  of  high  performance  teams,  where 
communication  overhead  is  very  less  because  the  participants  have  access  to  the 
information  without  asking  explicitly  [28].  Communication  overhead  is  typically  the  cost  of 
communication  and/or  interaction  measured  in  time,  internet  bandwidth  etc  [29].  Another 
aspect  that  vitally  affects  an  individual’s  ability  to  work  in  a  team  is  shared  mental  models. 
As  team  members  engage  in  a  group  activity,  they  tend  to  have  similar  thoughts/ideas  in 
order  to  accomplish  the  task  which  ultimately  results  in  less  communication  across  the 
team  [30].  These  aspects  are  essentially  a  part  of  team  dynamics  which  is  important  to  be 
considered  in  a  design  phase  of  any  experimental  groupware  activity. 

A  competitive  score  is  an  important  factor  in  motivating  participation.  Toups  Z  et  al 
observed  that  if  points  are  given  based  on  team  efforts,  participants  try  harder  to  work  as  a 
team  and  accomplish  the  task  in  a  well  coordinated  and  organized  team  effort  [31]. 

Advanced  Cardiac  Life  Support  (ACLS)  is  a  time-critical  activity  carried  out  by  a  dedicated 
high  performance  team.  Training  for  such  high  performance  teams  is  in  real  life  scenarios  is 
neither  possible  nor  advisable  since  it  is  generally  a  matter  of  a  person’s  life,  and 
simulation  training  is  one  of  the  best  possible  solutions  available.  According  to  Wayne  et  al, 
simulator  training  has  shown  significant  performance  improvement  in  a  team  of  physicians 
while  performing  ACLS  [32]. 

Training  in  Virtual  Worlds 

Based  on  their  purpose,  Collaborative  Virtual  Environments  [CVEs]  or  virtual  worlds  can 
be  categorized  into  one  of  the  following  types:  gaming,  socializing  or  online  community 
building,  and  educational  or  working  environments  [19].  [19,  20]  outline  the  various 
factors  that  need  to  be  present  in  a  virtual  world  to  be  suitable  for  educational  purpose. 
The  authors  compare  various  CVEs  and  come  to  the  conclusion  that  selection  of  a  particular 
CVE  depends  on  the  purpose  of  the  training  system.  Below,  we  will  briefly  explain  the 
research  on  CVEs  that  focus  on  healthcare  and  emergency  training. 
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Wiecha  et  al  explored  the  potential  of  a  virtual  world,  Second  Life  (SL),  as  a  delivering  tool 
for  continuing  medical  education  (CME)  [10].  In  their  study,  participants  had  to  select  and 
adjust  insulin  level  for  patients  with  type  2  diabetes.  For  that  purpose,  participants  had  to 
listen  to  an  instructional  40-minute  insulin  therapy  talk.  Two  mock  patients  are  also 
included  in  the  study  so  that  the  participants  can  interact  with  the  patients,  and  discuss 
within  themselves.  A  questionnaire  was  provided  to  the  participants  before  and  after  the 
talk  session.  The  study  shows  that  virtual  world  is  very  helpful  for  CME  education  by 
showing  significant  increase  in  the  score  after  the  talk  than  prior  to  it. 

Losh  [15]  lists  several  research  work  done  by  the  Interactive  Media  Laboratory  at 
Dartmouth  Medical  School  based  on  virtual  environments.  Virtual  Clinic  is  one  of  such 
work  where  a  virtual  clinic  is  designed  by  following  the  master  floor  plan.  The  main 
objective  of  this  work  is  to  allow  learners  to  learn  about  social  behavior  and  various 
procedures  in  clinical  environments.  The  Virtual  Terrorism  Response  Academy  (VRTA)  is  a 
simulation  based  game  to  train  users  on  how  to  act  during  crisis.  The  simulation  focuses  on 
providing  rescue  efforts  when  hazardous  materials  are  involved.  Before  starting  the  game, 
users  have  to  choose  and  assign  themselves  a  'role'.  Based  on  the  role,  which  can  be  a 
fireman,  emergency  medical  technician,  etc.,  training  is  provided  in  didactic  learning  space. 
Quizzes  and  interactive  videos  are  also  included  in  order  to  engage  the  users.  In  an 
experimental  session,  a  scenario  is  provided  to  the  users  and  the  main  objective  of  the 
users  is  to  practice  with  radiation  meters  and  see  how  the  exposure  levels  change  when 
nearing  hazardous  objects. 

Similar  to  VRTA  is  Play2Train  [18].  It  is  a  virtual  hospital  and  town  environment  which  is 
created  by  Idaho  Bioterrorism  Awareness  and  Preparedness  Program  (IDAPP).  The 
realistic  virtual  environment  of  Play2Train  provides  various  kinds  of  emergency 
preparedness  videos  in  virtual  classrooms,  and  also  supplements  several  training  exercises 
to  prepare  users  in  case  of  emergency  situations.  After  the  practice  sessions,  the  procedure 
followed  by  the  students  can  be  debriefed  by  the  instructor  to  clarify  the  experiences;  an 
essential  part  of  simulation-based  training. 

Callaghan  et  al  use  Second  Life  to  create  a  virtual  learning  environment  for  engineering 
education.  They  demonstrate  various  interactive  simulations  that  are  part  of  engineering 
education  [12].  Apart  from  the  simulations,  a  virtual  lecture  theater  is  also  present  in  the 
virtual  world  which  contains  interactive  mini/main  lecture  slideshow  viewer,  media  centre 
for  streaming  video  content  and  message  centers  for  feedback.  As  Second  Life  does  not 
provide  SDK,  the  authors  use  open  source  e-learning  software  SLOODLE  that  links  Second 
Life  with  a  course  management  tool  named  Moodle.  After  demonstration  of  the  simulations, 
the  participants  are  asked  questions:  if  they  answer  it  incorrectly,  they  have  to  run  the 
simulation  again  and  answer  the  questions  correctly. 

However,  this  study  lacks  the  assessment  and  the  evaluation  of  the  participants  and  they 
mention  that  these  shortcomings  will  be  their  main  focus  in  the  future.  Boulos, 
Hetherington,  and  Wheeler  [16]  describe  the  potential  use  of  Second  Life  in  medical  and 
health  education.  The  authors  provide  two  scenarios  -  'Virtual  Neurological  Education 
Centre’[VNEC,  (http://www.vnec.co.uk)  and  'Healthlnfo  Island’ 
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(http://infoisland.org/health_info).  The  former  demonstrates  a  scenario  where  users  are 
exposed  to  most  common  neurological  disability  symptoms.  Apart  from  the  symptoms, 
they  are  also  provided  with  related  information,  events,  and  facilities  in  the  Second  Life. 
The  latter  involves  providing  training  programs  for  virtual  communities.  It  also  intends  to 
provide  support  to  Second  Life  residents  by  providing  them  opportunities  to  participate  in 
different  medical  groups  dealing  with  stroke  support,  cerebral  palsy  etc. 

The  research  study  performed  by  Chodos  et  al  [1]  focuses  on  the  development  of  a  research 
based  virtual  environment  to  enhance  communication  skills  for  health  science  education. 
They  provide  two  case  studies.  The  first  one  is  the  development  of  EMT/ER  training 
simulation,  which  delivers  an  environment  to  train  EMT/ER  personnel  on  taking  care  of 
accident  victim  before  taking  him  to  a  hospital.  This  case  also  focuses  on  exchange  of 
patient  information  between  EMT  and  ER  personnel.  The  second  case  is  designed  to  teach 
various  competencies  to  students  like  rehabilitation  medicine,  nutrition,  physical  education 
etc.  For  the  second  case,  the  authors  design  a  simulation  in  order  to  increase 
communication  between  the  students  to  develop  a  home-care  plan  for  elderly  patient. 
Based  on  the  case  studies,  they  discuss  the  expectations  of  students  towards  virtual  world 
based  learning  and  the  quality  of  learning. 

There  are  several  other  projects  that  focus  on  virtual  healthcare  system.  Second  Health  is 
one  of  such  projects  where  users  can  learn  about  how  to  use  medical  devices  in  hospital 
settings  [12].  An  interactive  clinical  scenario  is  provided  to  learn  medical  device  training  in 
simulated  clinical  environment.  The  participants  are  provided  with  both  formative  and 
summative  feedback  during  the  training  session.  However,  the  system  does  not  provide 
clinical-skills  training  component  in  a  collaborative  environment  where  multiple  users 
make  a  team  and  perform  a  collaborative  task.  Similarly,  the  Ann  Myers  Medical  Centre 
[13]  and  the  nursing  training  program  from  Duke  University  [14]  provide  meeting  places 
for  medical  educators  and  students,  where  instructors  can  present  lectures  and  present 
educational  materials,  and  students  can  interact  with  each  other. 

Persuasive  Technologies 

Various  researchers  have  worked  on  finding  appropriate  way  to  persuade  users  to  perform 
various  activities.  Fogg  [8]  defines  persuasive  technologies  as  "interactive  computing 
systems  designed  to  change  people’s  attitudes  and  behaviors".  He  lists  various  persuasive 
technology  tools  (terminologies)  that  can  be  an  integral  part  of  any  system  in  order  to 
either  encourage  or  discourage  users  to  perform  some  actions  within  the  system  and 
change  their  attitude  and/or  behavior  while  doing  so.  In  medical  training/education, 
persuasion  is  one  of  the  most  important  factors  that  can  affect  the  performance  of 
trainees/students.  Use  of  meaningful  persuasive  components  (rewards,  realism,  social 
presence  etc)  enhances  the  learning  where  as  bad  design  of  persuasive  components 
hinders  it.  In  this  section,  we  will  mention  some  of  the  research  work  that  has  been  done  to 
encourage  users  to  perform  activities  within  a  given  system. 

Conradi  et.  al.  [17]  propose  an  idea  of  collaborative  learning  through  problem-based 
learning  (PBL)  in  Second  Life,  which  they  call  PREVIEW.  Researchers  prepared  five  virtual 
patient  scenarios  for  learners,  which  were  later  delivered  to  the  learners  through  Second 
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Life  platform.  The  main  objective  of  the  study  was  to  find  whether  computerized 
simulation  based  PBL  can  be  more  effective  than  classroom  based  PBL.  To  engage  students 
effectively  in  training  the  environment  provided  greater  realism,  active  decision  making, 
and  suitable  collaboration  environment  where  the  participants  can  interact  with  each 
other.  The  study  shows  that  realism,  and  suitable  interaction  environment  provided  by 
Second  Life  engages  students  effectively  in  learning. 

Consolvo  et  al  look  at  the  design  requirements  for  technology  to  encourage  physical 
activity  in  [21].  For  this  study,  they  come  up  with  a  mobile  phone  application  to  encourage 
users  to  perform  physical  activity.  The  application  has  three  different  versions:  baseline, 
personal,  and  sharing.  The  sharing  version  was  the  most  advanced  where  users  not  only 
can  see  their  activity,  but  also  can  share  their  performance  to  others  and  view  others 
performance.  Based  on  their  study,  they  describe  various  factors  that  motivate  users  to 
perform  physical  activity.  Giving  proper  credit  on  completion  of  each  task,  and  providing 
personal  awareness  on  users’  past  performance,  and  current  performance  are  the  basic 
elements  of  the  system  that  persuaded  users.  Another  important  factor  is  social  interaction. 
According  to  the  authors,  social  influence  creates  social  pressure,  which  motivates  users  to 
be  the  best  (or  at  least  not  the  worst)  in  the  society.  TripleBeat  [22]  is  also  a  similar  kind  of 
mobile  phone  based  system  that  motivate  runners  to  achieve  predefined  exercise  goals 
using  musical  feedback  as  well  as  competition  based  persuasion,  and  real-time  personal 
awareness.  The  experiment  results  conclude  that  the  system  is  “significantly  more 
effective"  in  helping  runners  to  achieve  the  goals. 

How  blogs  and  podcasts  can  be  helpful  tools  to  provide  more  sense  of  community  in  a 
group  is  explained  by  Firpo  et  al  [23].  The  major  objective  of  their  study  is  to  change 
attitude  and  behavior  of  a  community  at  School  of  Information  Systems  and  Technology 
(SISAT)  in  order  to  foster  a  sense  of  community  amongst  its  members.  Based  on  the 
functional  triad  explained  in  [8],  the  authors  conclude  that  social  presence  and  credibility 
as  the  key  factors  to  persuade  the  members  in  the  community. 

Several  virtual  reality  based  games  have  already  evolved  to  motivate  users  to  maintain 
good  health.  The  following  simulation  based  applications  have  proved  the  fact  that 
simulated  environments  are  very  effective  to  change  one’s  attitude  and  behavior.  The 
Tetrix  VR  Bike  [24]  is  an  environmental  simulation  that  motivates  users  to  work  out  on  this 
device  by  exploring  the  virtual  environment.  The  faster  users  pedal,  the  faster  will  be  the 
exploration.  Another  simulated  environment  is  Bronkie  the  Bronchiasaurus  [25],  which  is 
designed  to  help  kids  with  asthma  to  manage  their  condition.  The  study  showed  that  the 
asthmatic  children  who  played  the  game  for  at  least  30  minutes  report  increased  self- 
efficacy  to  take  care  of  their  chronic  condition.  Similarly,  HIV  Roulette  [26]  is  another 
simulation  to  provide  immediate  insights  into  sexual  behavior.  Users  can  view  and  select 
hypothetical  character  along  with  gender  and  behavior.  Based  on  the  selection  criteria,  the 
system  reports  whether  the  specified  behavior  is  likely  to  cause  HIV  or  any  other  sexually 
transmitted  diseases. 
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Conceptual  Design 

The  main  objective  of  this  system  is  to  allow  users  to  access  the  system  from  virtually 
anywhere  [in  presence  of  internet  connectivity].  The  figure  displays  real  world  and  virtual 
world  sites.  Although  the  team  members  are  in  the  same  virtual  world  location,  they 
actually  are  logged  in  from  different  locations  in  the  real  world.  The  person  who  is 
responsible  for  doing  CPR  has  access  to  the  haptic  device. 

Design  and  Development  of  Collaborative  Virtual  Environment 

Figure  1  shows  the  design  of  the  system  from  the  perspective  of  a  user  who  is  performing 
the  cardio  pulmonary  resuscitation  [CPR]  action.  The  system  can  be  divided  into  four 
components:  collaborative  virtual  environment  [CVE]  component,  haptic  component,  voice 
component,  and  database  component. 
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Figure  1.  System  Design  Overview 

Using  Active  Worlds®  [AW]  as  our  CVE,  we  developed  a  virtual  hospital  which  allowed  for 
multiple  to  log  on  simultaneously  and  practice  their  ACLS  skills.  Users  meet  as  a  team  and 
are  capable  of  performing  various  actions  involved  in  a  code  scenario.  This  includes 
providing  CPR,  assisting  in  breathing,  evaluating  patient  vitals  and  providing  the  necessary 
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medications.  Avatars,  objects  and  animations  necessary  to  support  these  interactions  were 
developed  and  integrated  into  the  environment.  We  also  integrated  persuasive  elements 
into  the  CVE.  These  elements  included  timely  alerts  and  scores  for  formative  feedback.  One 
of  the  key  components  for  ensuring  skills  training  was  haptics-based  CPR  implementation. 
The  haptic  component  is  responsible  for  measurement  of  the  CPR  rate  during  training, 
while  proving  realistic  force  feedback  of  the  recoil  during  compressions.  We  used  the 
Novint  Falcon®  haptic  joystick  for  this  implementation. 

The  users  were  able  to  communicate  to  each  other  via  Skype®.  Skype  was  found  to  be  an 
appropriate  conduit  as  it  provided  the  necessary  quality  required,  while  supporting  two- 
way  communication  between  multiple  parties.  We  also  integrated  this  environment  with  a 
database  for  tracking  users,  their  actions  performance  though  the  various  scenarios.  Figure 
2  shows  a  snap  shot  of  the  CVE  from  the  perspective  of  the  user. 


Figure  2.  Virtual  ACLS  training  room  with  required  objects 


The  quality  of  performance  of  a  team  performing  the  ACLS  procedure  is  highly  dependent 
on  the  time  taken  to  perform  each  individual  task.  Consequently,  our  scoring  system  is 
based  on  the  time  taken  to  complete  each  sub-task  of  the  ACLS  protocol.  This  was  verified 
by  experienced  ACLS  instructors.  Users  were  provided  with  a  rating  between  0  and  100 
that  was  estimated  based  on  the  ratio  points  awarded  to  the  total  possible  points  for  the 
given  scenario.  This  scoring  system  was  consistent  for  the  virtual  world  training  and 
testing,  and  also  the  actual  testing  at  the  real  training/testing  center. 

Recent  Results  of  Pilot  Study 

We  conducted  a  preliminary  experiment  with  24  participants,  who  were  novices  to  ACLS. 
We  divided  the  participants  into  six  groups  of  four.  The  participants  were  provided  with 
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didactic  instruction  on  ACLS  and  scenarios  they  will  be  tested  on.  We  asked  each 
participant  to  take  a  quiz,  with  10  multiple  choice  questions  related  to  ACLS,  prior  to 
conducting  the  experiment,  in  order  to  verify  that  the  participants  are  at  the  same  level  of 
expertise  on  ACLS.  Table  1  depicts  the  distribution  of  the  participants  into  control  and 
experiment  groups. 


Group 

Didactic 

Instruction 

Persuasive  CVE 
Training 

Non-Persuasive 
CVE  Training 

Count 

Control  (C) 

X 

2 

Experiment  1  (P) 

X 

X 

2 

Experiment  2 

(NP) 

X 

X 

2 

Table  1.  Participant  Distribution  (X  indicates  training  received) 


The  effect  of  the  training  received  was  tested  via  a  skills  transfer  experiment  in  a 
simulation  environment.  Participants  were  tested  in  groups  (same  groups  as  training)  on  a 
code  simulated  using  a  manikin.  Certified  ACLS  trainers  evaluated  the  performance  of  the 
groups. 


Figure  3  depicts  the  final  performance  of  the  various  groups.  The  experiment  groups 
(persuasive  and  non-persuasive  CVE  training)  were  given  significantly  higher  scores 
compared  to  the  control  groups.  It  should  also  be  noted  that  while  the  results  were  not 
significant,  groups  training  in  persuasive  CVEs  were  awarded  higher  scores,  on  average. 
This  provides  evidence  to  support  further  exploration  of  the  merits  of  CVE  training 
segmented  with  persuasive  elements. 


Performance  in  the  real  training/testing  center  (Vfib)  Performance  in  the  real  trainingttesting  center  (PEA) 


ACLS  teams  ACLS  teams 

Figure  3.  Performance  of  the  groups  in  the  actual  training  room  at  real  ACLS 
training/testing  center  (Cl  &  C2:  control  groups  1  and  2;  PI  &  P2:  persuasive  groups 
1  and  2;  NP1  &  NP2:  non-persuasive  groups  1  and  2) 
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The  purpose  of  the  proof  of  concept  study  was  to  provide  evidence  that  a  CVE  would 
benefit  team-training  in  critical  care  scenarios.  In  order  to  explore  this  further,  the  size  of 
the  study  needs  to  be  increased.  In  addition,  the  study  will  need  to  include  health  care 
professionals  so  as  to  focus  on  team  training,  as  opposed  to  implementation  of  protocols 
learned  through  didactic  training. 


Key  Research  Accomplishments 

1.  Development  of  virtual  world  which  is  based  on  front  of  code  cart,  available 
medications  in  this  hospital,  and  EKG  monitor. 

2.  Development  of  clinical  scenarios  for  ACLS  in  the  virtual  world 

3.  Linking  of  a  haptic  devices  to  a  virtual  world  for  CPR  training 

4.  Validation  of  the  CPR  training  module 

5.  Development  of  the  persuasive  framework  in  virtual  worlds 

6.  Cloud  based  reporting  system  for  aggregation  and  archiving  of  data 

7.  Validation  of  Virtual  World  based  training  simulation  for  ACLS  (Pilot  study.) 

8.  Validation  of  persuasive  framework  in  virtual  worlds. 


Reportable  Outcomes: 

See  Appendix  I:  K  Kahol,  M  Vankipuram,  V  Patel,  M  Smith,  "Deviations  from  Protocol  in  a 
complex  Trauma  environment:  Errors  or  innovations?",  lournal  of  Biomedical  Informatics. 
vol  44,  425-431,2011 

See  Appendix  II:  P  Khanal,  S  Parab,  K  Kahol,  Mark  Smith;  Collaborative,  Time-Critical,  Multi- 
Sensory  Training  in  Virtual  Worlds  with  Persuasive  Elements;  Submitted  to  Computer 
Human  Interaction  fCHD.  2011 


Conclusion  and  Next  Steps: 

Essentially  all  of  the  development  of  the  virtual  learning  environments  have  been 
completed.  And  while  Worlds  (AW)  provides  an  environment  to  create  our  own  virtual 
world,  we  have  found  that  the  interaction  provided  by  AW  is  limited.  Realistic  animation  of 
key  activities  is  critical  to  situational  awareness.  The  framework  of  active  worlds  has  made 
it  difficult  to  achieve  the  realism  required  to  meet  the  needs  of  the  preferred  environment. 
Porting  the  existing  simulation  to  a  virtual  world  developed  with  the  Unreal  Development 
Toolkit  (UDK)  can  enable  us  to  achieve  this  goal.  A  UDK-based  environment  provides  (i) 
ability  to  create  dynamic  and  high  fidelity  environments,  (ii)  animations,  character  models, 
and  ambient  sounds  that  allow  for  an  immersive  simulation  experience  and  (iii)  the  ability 
to  create  and  run  servers  on  our  own,  thereby  ensuring  privacy  of  data  captured  through 
the  simulation.  Development  of  the  CVE  using  UDK  will  be  the  next  step  in  this  work  while 
waiting  the  approval  of  the  final  IRB. 
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Communication  is  an  integral  part  of  team-training.  Therefore,  in  addition  to  creating  a 
more  realistic  environment,  we  will  be  analyzing  conversations  between  team  members 
quantitatively.  Voice  communication  between  members  can  be  assessed  by  integrating 
TeamSpeak  SDK®  with  UDK.  Such  analysis  can  enable  us  to  pool  metrics  such  as  duration, 
frequency,  directionality  and  urgency  of  communication. 

Through  our  preliminary  work  we  have  shown  that  CVE  benefit  training  for  2  types  of 
cases  (VFIB  and  PEA)  that  may  be  encountered.  In  order  to  generalize  the  applicability  of 
the  CVE,  additional  scenarios,  such  as,  bradycardia,  and  trauma  cases  may  also  be  made 
available.  In  the  original  grant,  CVE  were  proposed  as  a  viable  team-training  supplement 
for  critical  care  scenarios.  Similar  to  ACLS,  the  Advance  Trauma  Life  Support  guideline  is 
used  to  guide  clinical  teams  in  the  treatment  of  Trauma  cases.  Team  training  and 
communication  is  critical  to  outcomes  in  Trauma.  In  order  to  show  the  generalizability  of 
the  CVE,  a  trauma  scenario  will  be  implemented  as  well.  The  implementation,  testing  and 
validation  of  a  scenario  in  Trauma  will  enable  establishing  the  ability  of  CVEs  to  provide 
training  for  any  critical  care  scenario  requiring  team  interactions. 
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Protocol  standardizations  are  important  for  consistent  and  safe  practices.  However,  complex  clinical 
environments  are  highly  dynamic  in  nature  and  often  require  clinicians,  confronted  with  non-standard 
situations,  to  adjust  and  deviate  from  standard  protocol.  Some  of  these  deviations  are  errors  which  can 
result  in  harmful  outcomes.  On  the  other  hand,  some  of  the  deviations  can  be  innovations,  which  are 
dynamic  adjustments  to  the  protocols  made  by  people  to  adapt  the  current  operational  conditions  and 
achieve  high  accuracy  and  efficiency.  However,  there  is  very  little  known  about  the  underlying  cognitive 
processes  that  are  related  to  errors  and  innovations.  In  this  study  we  investigate  the  extent  to  which  devi¬ 
ations  are  classified  as  errors  or  innovations,  as  a  function  of  expertise  in  a  Trauma  setting.  Field  obser¬ 
vations  were  conducted  in  a  Level  1  Trauma  unit.  A  total  of  10  Trauma  cases  were  observed  and  collected 
data  was  analyzed  using  measures  that  included  customized  activity-error-innovation  ontology,  time- 
stamps  and  expertise  of  the  team  members.  The  results  show  that  expertise  of  the  caregivers  and  criti¬ 
cality  of  a  patient’s  condition  in  critical  care  environment  influence  the  number  and  type  of  deviations 
from  standard  protocol.  Experts’  deviations  were  a  combination  of  errors  and  innovations;  whereas  the 
novices’  deviations  were  mostly  errors.  This  research  suggests  that  a  novel  approach  must  be  taken  into 
consideration  for  the  design  of  protocols  (including  standards)  and  compliance  measurements  in  com¬ 
plex  clinical  environments. 

©  201 1  Elsevier  Inc.  All  rights  reserved. 


1.  Introduction 

Healthcare  systems  are  complex  systems  with  non-linear  inter¬ 
actions  and  dynamic  emergent  behavior  [1].  From  the  initial  days 
of  simple  doctor-patient  relationship,  healthcare  today  has  ex¬ 
panded  to  include  a  multitude  of  factors  that  increase  the  complex¬ 
ity  of  the  system.  This  is  true  at  various  levels  of  healthcare 
wherein  multitude  of  people  interact  with  other  people  and  in  re¬ 
cent  times  a  myriad  of  complex  technology.  The  presence  of  such 
dense  and  interrelated  network  structure  of  interactions  between 
these  entities  makes  operations  in  complex  networks  often  intrac¬ 
table.  This  can  be  seen  when  tracking  activities  and  workflow  in 
critical  care  environments  such  as  emergency  departments  and 
Trauma  centers.  From  an  intervention  perspective,  the  issue  of 
intractability  makes  design,  implementation  and  evaluation  of 
the  intervention  difficult.  Poorly  implemented  interventions  could 
adversely  affect  patient  safety.  Consequently,  interventions  in 
complex  environments  need  to  be  understood  at  a  fundamental 
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level  to  ascertain  how  to  successfully  implement  interventions 
and  ensure  that  these  interventions  will  improve  patient  safety. 

A  class  of  interventions  that  has  proven  to  be  very  useful  in 
complex  environments  is  protocols.  Protocols  serve  as  a  means  to 
accomplish  complex  tasks  by  dividing  them  into  simpler  observa¬ 
ble  units.  Typically,  protocols  suggest  a  sequence  of  these  atomic 
tasks  and  define  the  criteria  for  success.  Most  clinical  procedures 
can  involve  several  steps  and  having  a  protocol  helps  in  standard¬ 
izing  the  steps  and  ensuring  that  all  steps  are  completed.  The  util¬ 
ity  of  protocols  is  assessed  using  checklists,  a  tool  that  has  proven 
to  be  a  very  effective  in  the  management  and  control  of  processes 
in  complex  environments  [2].  Checklists  help  in  several  ways  to 
ensure  quality  and  safety  and  have  become  an  easily  implement- 
able  method  to  avoid  errors.  Duane  et  al.  [3]  assessed  the  effect 
of  a  protocol  for  Central  Venous  Line  (CVL)  placement  on  blood 
stream  infections  (BSI)  and  patient  outcome  in  a  Trauma  Intensive 
Care  Unit  (ICU).  It  was  found  that  the  protocol,  when  supported  by 
a  nursing  checklist,  reduced  BSI  incidence  rates  and  minimized  the 
length  of  stay  in  the  Trauma  ICU.  In  addition  to  having  a  positive 
impact  on  clinical  outcomes,  protocols  and  checklists  aid  in  reduc¬ 
ing  costs  incurred  by  the  clinical  unit.  Semel  et  al.  [4]  performed  a 
decision  analysis  of  the  implementation  of  a  protocol  checklist  in  a 
US  hospital  for  a  1  year  time  period.  It  was  found  that  checklist 
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implementation  could  generate  cost  savings  after  enabling  the 
avoidance  of  five  major  complications  (assuming  a  baseline  com¬ 
plication  rate  of  3%).  Protocols  and  checklists  enable  institutions 
to  reduce  costs  by  avoiding  expensive  medical  errors  and  conse¬ 
quently  improving  the  quality  of  patient  care. 

Agencies,  such  as  the  American  College  of  Surgeons  (ACS),  the 
Society  of  American  Gastrointestinal  and  Endoscopic  Surgeons 
(SAGES)  and  the  American  Board  of  Surgery  (ABS)  have  recognized 
the  importance  of  protocols  and  standardizations.  ABS,  for  exam¬ 
ple,  have  made  training  in  Advanced  Cardiovascular  Life  Support 
(ACLS)  and  Advanced  Trauma  Life  Support®  (ATLS®)  protocols 
mandatory  for  general  surgery  certification  [5].  In  addition,  recent 
initiatives  by  the  Department  of  Human  Health  Services  have  sup¬ 
ported  efforts  on  standardizations  and  the  use  of  information  tech¬ 
nology  to  develop  protocols  [6]. 

Much  research  on  medical  errors  attempts  to  identify  error  as 
deviations  from  some  known  standard  [7-9].  While  an  error  being 
interpreted  as  a  deviation  may  be  true,  the  converse  need  not  nec¬ 
essarily  be  accurate.  In  fact,  it  is  possible  that  a  deviation  from  a 
protocol  may  be  an  innovation  designed  to  maximize  patient 
safety.  The  identification  of  such  cases  is  critical  to  the  evaluation 
and  improvement  of  existing  protocols.  In  addition  to  protocol 
management,  it  is  important  for  novice  clinicians  to  identify  such 
cases  and  adapt  existing  protocols  to  the  situation  at  hand.  While 
understanding  the  importance  of  standards  is  part  of  good  clinical 
practice  and  should  be  grasped  effectively,  knowing  when  to  devi¬ 
ate  from  the  protocol  can  indicate  flexibility  and  adaptability  that 
is  important  in  assuring  good  and  safe  decisions.  It  can  accordingly 
be  problematic  if  our  education  system  and  our  management 
structures  advocate  following  standard  protocols  alone,  failing  to 
acknowledge  that  students  also  need  to  learn  how  to  handle  com¬ 
plex  problem  solving  that  is  outside  the  boundaries  of  “standard 
solutions”.  A  example  of  such  a  complex  problem  is  that  of  the 
emergency  landing  of  US  Airways  flight  1549  in  the  Hudson  River 
-  a  situation  in  which  the  pilots  made  good  decisions  about  follow¬ 
ing  some  protocols  but  departing  from  others  [10].  In  this  paper, 
we  seek  to  explore  the  relation  between  errors,  innovations,  proto¬ 
cols  and  expertise  in  complex  critical  care  environments. 

From  a  cognitive  perspective,  error,  innovation  and  effectiveness 
of  protocol  is  intimately  linked  with  expertise  of  the  clinicians.  Patel 
et  al.  studied  the  relationship  between  task  difficulty  and  expertise 
[11].  The  authors  employed  semantic  analysis  and  found  that  ex¬ 
perts  were  able  to  use  well  developed  knowledge  base  and  superior 
reasoning  strategies  in  clinical  reasoning.  Groen  and  Patel  [12]  in 
another  publication  isolated  the  reasoning  process  that  physicians 
go  through  when  diagnosing  a  clinical  case,  using  techniques  to 
identify  knowledge  structures.  They  showed  that  in  medicine,  ex¬ 
perts  tend  to  follow  a  top-down  reasoning  strategy  wherein  reason¬ 
ing  from  a  hypothesis  to  account  for  the  case  data,  which  seemed 
anomalous  when  compared  to  other  domains.  This  is  an  important 
finding  from  the  perspective  of  studying  errors  and  innovations.  In 
other  domains  wherein  experts  tend  to  gather  data  and  assemble 
hypothesis,  there  is  scope  for  significant  amount  of  trial  and  errors. 
On  the  other  hand,  in  clinical  decision  making,  experts  more  often 
than  not  utilize  a  top-down  approach  to  decision  making.  It  has 
been  shown  that  this  methodology  when  combined  with  experi¬ 
ence  driven  cognitive  constructs  results  in  experts  making  fewer  er¬ 
rors  compared  to  novices.  It  is  plausible  that  when  experts  do 
deviate,  they  are  more  likely  to  be  innovations. 

Another  aspect  of  cognition  that  needs  to  be  accounted  for  is  the 
capability  of  a  clinician  to  generalize  given  data  into  correct  diag¬ 
nosis.  Cognitive  research  in  medicine  [13]  has  shown  that  clini¬ 
cians  can  generate  different  levels  of  mental  representations, 
from  the  very  specific  to  the  very  general.  The  critical  factor  in 
determining  generality  is  typically  the  degree  of  high  level  exper¬ 
tise  of  the  clinician,  namely,  specialized  or  specific  expertise  (i.e., 


knowledge  of  a  particular  sub-domain  of  medicine,  such  as  endo¬ 
crinology  or  cardiology).  Higher-level  representations  are  gener¬ 
ated  by  these  more  expert  clinicians,  whereas  lower-level  and 
more  detailed  representations  are  typically  generated  by  novices, 
or  more  commonly,  intermediate  level  clinicians  (e.g.,  senior  med¬ 
ical  students,  recent  graduates,  and  residents). 

This  condition  points  to  the  ability  of  experts  to  apply  generic 
rules  to  a  given  case,  giving  them  extra  cognitive  resources  to  apply 
innovations  and  limit  errors.  Research  has  shown  that  experts  as  a 
result  of  their  practice,  learn  to  associate  individual  items  in  work¬ 
ing  memory  with  the  contents  in  long  term  memory,  which  result 
in  the  development  of  conceptual  organizations  in  memory  called 
retrieval  structures  [14].  An  expert  can  use  these  retrieval  struc¬ 
tures  to  provide  selective  and  rapid  access  to  long  term  memory. 
On  the  other  hand  novices  seem  to  occupy  their  working  memory 
and  long  term  memory  resources  in  the  details  of  the  case  (due  to 
the  lack  of  mature  retrieval  structures)  which  may  be  irrelevant.  In 
such  type  of  workload,  it  may  be  challenging  to  innovate  and 
depending  on  the  workload,  one  may  make  extensive  errors  as  is 
in  the  case  of  complex  environments.  In  fact,  research  confirms 
that  a  key  element  of  retrieval  structures  is  their  use  by  experts 
to  eliminate  irrelevant  information  [15]  freeing  working  memory 
for  innovative  thinking. 

In  general  the  literature  on  clinical  expertise,  gives  clues  into 
the  underlying  mechanisms  of  the  relationship  between  errors 
and  innovations.  One  area  of  research  that  has  explored  the  mech¬ 
anisms  of  innovations  is  cognitive  basis  of  creativity  [16].  This  field 
explores  the  cognitive  basis  underlying  creative  thinking  and  rea¬ 
soning.  It  identifies  conditions  that  lead  to  creation  and  innovation 
and  is  based  on  the  hypothesis  that  creativity  is  supported  by  pre¬ 
invention  structures  and  the  explanation  structures  in  experts.  This 
is  a  very  intriguing  model  for  creativity  and  cognition  but  its  rele¬ 
vance  to  complex  domains  such  as  Trauma  may  be  limited.  In  gen¬ 
eral,  the  theories  from  creativity  tend  to  focus  on  free  thinking 
approach  wherein  timeliness  of  creativity  is  not  a  big  factor.  On 
the  other  hand,  in  complex  environments  such  as  Trauma  or  criti¬ 
cal  care,  timeliness  of  decision  making  may  fundamentally  alter 
the  innovation  process  and  it  is  important  to  study  the  mecha¬ 
nisms  underlying  errors  and  innovations  separately. 

The  present  research,  to  our  knowledge,  is  one  of  the  first  to 
study  to  examine  the  cognitive  basis  of  innovation  mechanisms 
in  experts  in  medicine.  The  following  section  provides  the  required 
background  for  understanding  the  concepts  of  innovation  (and  er¬ 
rors)  and  their  classification. 

2.  Analytic  framework 

Fig.  1  describes  a  hierarchical  schema  for  deviation  classifica¬ 
tion  in  Trauma.  This  schema  was  developed  based  on  field  observa- 


Fig.  1.  Classification  of  deviations  in  Trauma  based  on  observation  data  from  [17]. 
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tions  done  in  December  2009  at  the  Level  1  Trauma  center  at  Ban¬ 
ner  Good  Samaritan  Hospital  in  Phoenix,  Arizona.  Each  of  the  key 
components  of  the  classification  is  discussed  below. 

2.1.  Identification  of  deviations 

Deviations  could  be  broadly  defined  as  steps  performed  that  are 
not  on  an  accepted  pre-defined  protocol.  For  the  analysis  of  devia¬ 
tions  in  Trauma,  the  most  appropriate  guideline  or  protocol  avail¬ 
able  is  the  ATLS  protocol  [17].  It  is  mandatory  that  this  protocol  be 
followed  in  every  Level  1  Trauma  center  for  accreditation  purposes. 
Research  has  shown  that  ATLS  protocol  is  a  very  effective  protocol 
in  improving  the  quality  of  care  in  Trauma  centers  across  USA  [18] 
and  is  overseen  by  the  American  College  of  Surgeons.  The  key  steps 
in  the  ATLS  protocol  are  as  follows  [17], 

(i)  Primary  survey  -  Assess  the  Airway,  Breathing  and 
Circulation  (ABC)  of  the  patient  and  secure  the  same.  Per¬ 
form  Disability  assessment  and  control  Exposure  and  the 
Environment.  This  is  the  ABCDE  of  Trauma  management. 

(ii)  Secondary  survey  -  Complete  a  detailed,  head-to-toe  exami¬ 
nation  and  obtain  AMPLE  (Allergies,  Medications,  Past  his¬ 
tory,  Last  eaten  information  and  Events  leading  to  Trauma) 
history  from  patient. 

(iii)  Definitive  diagnosis  and  management  -  Provide  a  treatment 
plan  and  discharge  or  transfer  patient  from  Trauma. 

Table  1  details  the  steps  (in  sequence)  of  the  protocol.  A  devia¬ 
tion  is  marked  if  a  step  that  is  on  the  ATLS  protocol  is  skipped,  an 
extra  step  is  performed  or  if  a  certain  task  is  performed  out  of  or¬ 
der.  Typical  deviations  from  the  protocol  include  failure  to  perform 
a  log  roll  (key  step  in  protecting  the  spine  during  assessment),  or  a 
resident  making  an  error  by  omitting  steps  or  in  some  cases,  add¬ 
ing  unnecessary  steps.  In  the  following  sub-sections,  we  provide 
definitions  for  the  various  types  of  deviations  that  were  observed 
in  this  study. 

2. 1.1.  Deviations  as  errors 

We  define  error  as  any  deviation  that  potentially  impacts  pa¬ 
tients  and  their  treatment  outcome  negatively.  Some  examples  of 
errors  detected  in  the  data  gathered  from  Trauma  include: 

•  Clinician  is  not  present  in  the  Trauma  room  when  the  patient 

arrives. 

Treatment  of  a  Trauma  patient  is  a  time  critical  activity  that  re¬ 
quires  preparation  for  efficient  implementation.  Delay  in  arriving 
for  the  Trauma  reduces  the  time  the  clinician  has  to  prepare  for 
the  Trauma  case.  Such  an  error  reduces  Trauma  efficiency  and,  in 
worst  case,  can  potentially  have  negative  clinical  outcomes. 

•  Clinician  staples  a  patient’s  wound  inaccurately  causing  the  cli¬ 
nician  to  redo  one  or  more  staples. 

Such  errors  in  psychomotor  performance  often  occur  due  to  the 
time  critical  and  expertise-driven  nature  of  the  complex  environ¬ 
ments.  Clinicians  may  make  such  errors  due  to  the  added  cognitive 
pressure.  These  type  of  errors  have  been  reported  in  literature  [19] 
and  lead  to  the  clinician  deviating  from  the  protocol  to  rectify  the 
error. 

•  The  Trauma  team  fails  to  perform  a  log  roll  when  examining  the 

spine  of  the  patient. 

The  log  roll  ensures  that  the  patient’s  cervical  neck  and  spine  is 
protected  during  secondary  assessment.  Failure  to  perform  the  log 


Table  1 

Key  steps  in  Initial  Assessment  and  Management  ATLS  Protocol  adapted. 

ATLS  -  Initial  Assessment  and  Management  Protocol 

(A)  Primary  survey  assessment  ofABCDE’s 

1 .  Airway  with  cervical  spine  protection 

2.  Breathing 

3.  Circulation  with  control  for  external  hemorrhage 

4.  Disability  with  brief  neurological  evaluation 

5.  Exposure/environment 

(B)  Resuscitation 

1 .  Oxygenation  and  ventilation 

2.  Shock  management,  intravenous  lines,  warmed  Ringer’s  lactate  solution 

3.  Management  of  life-threatening  problems  identified  in  the  primary  survey 
is  continued 

(C)  Adjuncts  to  primary  survey  and  resuscitation 

1.  Monitoring 

a.  Arterial  blood  gas  analysis  and  ventilator  rate 

b.  End-tidal  carbon  dioxide 

c.  Electrocardiograph 

d.  Pulse  oximetry 

e.  Blood  pressure 

2.  Urinary  and  gastric  catheters 

3.  X-rays  and  diagnostic  studies 

a.  Chest 

b.  Pelvis 

c.  C-spine 

d.  Diagnostic  peritoneal  lavage  (DPL)  or  abdominal  ultrasonography 

(D)  Secondary  survey,  total  patient  evaluation:  physical  examination  and  history 

1.  Head  and  skull 

2.  Maxillofacial 

3.  Neck 

4.  Chest 

5.  Abdomen 

6.  Perineum/Rectum/Vagina 

7.  Musculoskeletal 

8.  Complete  neurologic  examination 

9.  Tube  and  fingers  in  every  orifice 

(E)  Adjuncts  to  the  secondary  survey 

1.  Computerized  tomography 

2.  Contrast  X-ray  studies 

3.  Extremity  X-rays 

4.  Endoscopy  and  ultrasonography 

(F)  Definitive  care 

(G)  Transfer 


roll  could  potentially  compromise  the  patient’s  spine  and  nervous 
system.  Consequently,  it  is  considered  to  be  an  erroneous 
deviation. 

2.1.2.  Deviations  as  innovations 

Innovation  can  be  defined  as  a  deviation  from  the  protocol  that 
may  positively  affect  the  patient’s  outcome.  Innovations,  that  are 
properly  validated  and  generalize,  can  potentially  become  part  of 
the  protocol  that  it  initially  deviated  from.  Some  examples  of  inno¬ 
vation  include, 

•  The  clinician  prioritizes  secondary  examination  of  the  patient  to 
address  time  critical  aspects  of  the  patient’s  treatment. 

The  ATLS  protocol  allows  for  clinicians  to  adapt  the  processes 
to  a  specific  patient.  However,  there  are  times  when  clinicians 
need  to  deviate  even  within  the  broad  framework  to  care  for 
his/her  patient.  Penetrating  injuries  to  the  chest,  for  example, 
are  given  higher  priority  than  head  and  maxillofacial  examination 
during  the  head-to-toe  survey  of  the  patient  to  ensure  that  condi¬ 
tions  of  pneumothorax  or  hemothorax  (chest  cavity  compromised 
by  air  or  blood,  respectively)  are  detected  early.  This  type  of  inno¬ 
vation,  which  can  be  understood  as  dynamic  innovation,  is  quite 
common  but  presents  an  important  challenge  for  judging 
compliance. 
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•  An  attending  physician  suggests  that  the  patient’s  arm  could  be 
taped  up  as  they  anticipate  problems  a  patient  may  have  during 
a  required  X-ray  scan. 

This  is  an  anticipatory  innovation  wherein  an  expert  based  on 
previous  experience  can  predict  the  possible  outcomes  of  an  action 
and  can  provide  preventative  or  supportive  inputs. 

•  The  attending  physician  shares  an  innovative  method  for  a  pro¬ 
cedure  for  treating  the  patient  that  has  not  yet  been  validated. 

This  is  a  knowledge  based  innovation.  Experts  are  adept  at 
learning  from  new  sources  and  are  capable  of  carefully  testing 
new  procedures  and  innovations.  These  types  of  innovations  can 
potentially  be  dangerous  if  implemented  by  novices  but  experts 
can  devise  a  careful  plan,  roll  out  and  test  new  methods  in  a  con¬ 
trolled  manner. 

2 A. 3.  Proactive  and  reactive  procedural  deviations 

During  classification  of  deviations  based  on  preliminary  data 
gathered,  it  was  found  that  a  large  number  of  deviations  were  nei¬ 
ther  errors  nor  innovations  as  defined  above.  Some  examples 
include, 

•  Resident  pauses  when  conducting  the  primary  survey  in  order 
to  ask  the  patient  to  co-operate. 

•  The  Trauma  nurse,  reacting  to  a  patient  vomiting,  moves  over 
help  the  patient  clean  up. 

•  A  Trauma  nurse  anticipating  a  patients’  arrival,  requests  the 
Radiology  technician  to  insert  the  X-ray  apparatus  below  the 
patient’s  sheets,  prior  to  patient  arrival. 

All  three  cases  are  neither  errors,  nor  innovation  as  they  do  not 
directly  impact  patient  outcomes  but  rather  are  actions  demanded 
by  dynamic  nature  of  the  complex  environments.  The  first  two 
deviations  mentioned  above  are  examples  of  clinicians  performing 
procedural  steps  in  reaction  to  patient-specific  actions.  These  clas¬ 
ses  of  deviations  are  termed  Reactive  Procedural  Deviations.  The  last 
case  is  a  procedural  action  requested  proactively  by  the  Trauma 
nurse  to  improve  the  efficiency  of  the  Trauma  case.  Hence,  this 
class  of  deviations  is  called  Proactive  Procedural  Deviations. 

Using  the  analytic  framework  defined  above,  deviations  were 
identified  using  ATLS  protocol  for  “Initial  Assessment  and  Manage¬ 
ment”.  The  following  were  the  specific  questions  we  attempted  to 
answer  through  analysis  of  the  deviations, 

Question  l:  How  often  do  the  Trauma  team  members  deviate 
from  the  Advanced  Trauma  Life  Support  protocol? 

Question  2:  When  clinicians  deviate,  what  are  the  types  of  devi¬ 
ations  made? 

Question  3:  How  do  these  types  of  deviations  vary  with  the 
experience  (level  and  type)  of  the  members  of  the  clinical 
team? 

3.  Methods 

3A.  Site  description 

The  field  observations  for  this  work  were  conducted  in  Banner 
Good  Samaritan’s  Trauma  unit,  one  of  6  Level  1  Trauma  centers 
in  the  Phoenix  metropolitan  area.  Approximately  3000  patients 
are  treated  annually  in  this  five  bed  unit.  The  Trauma  center  has 
dedicated  hospital  resources  for  the  management  of  Trauma  pa¬ 
tients  throughout  all  aspects  of  care,  including  initial  evaluation 
and  resuscitation,  acute  care  and  rehabilitation.  In  addition,  the 
Trauma  unit  collaborates  with  surgeons  from  neurosurgery,  car- 


diothoracic,  vascular,  orthopedic,  plastics,  ophthalmology,  urology 
and  internal  medicine  departments  to  provide  the  required  care  for 
incoming  patients.  The  Trauma  team  (present  during  every  shift) 
includes  1  Trauma  resident,  2  Trauma  nurses,  1  Trauma  attending, 
1  anesthesiologist,  1-2  juniors  residents,  1-2  medical  students, 
and  radiology  and  lab  technicians. 

At  Banner  Good  Samaritan’s  Trauma  center,  patients  are  treated 
by  the  Trauma  team  with  the  resident  acting  as  the  Trauma  team  lea¬ 
der.  The  resident  treats  the  patient  under  minimal  supervision  of  the 
attending  Trauma  surgeon.  In  each  case,  out  of  the  two  Trauma 
nurses,  one  nurse  acts  as  the  primary  nurse  assisting  the  resident, 
while  the  other  Trauma  nurse  takes  charge  of  documenting  activi¬ 
ties.  Therefore  in  each  Trauma  is  dealt  with  a  core  team  that  includes 
1  PGY3/4  level  resident,  1  PGY1  \2  level  resident,  1  Trauma  attending, 
1  (primary)  Trauma  nurse  and  1  technician  (radiology). 

Trauma  nurses  supporting  the  Trauma  leader  are  experienced 
registered  nurses  (RNs)  with  5-10  years  of  critical  care  experience. 
A  total  of  36  residents  complete  a  2  month  Trauma  rotation  in  the 
3rd-4th  year  of  their  residency  program. 

3.2.  Study  description  and  methodology 

This  study  was  approved  by  the  Institutional  Review  Board  and 
the  informed  consents  were  obtained  from  the  participants  on  each 
encounter.  Field  observations  were  gathered  by  one  researcher  over 
a  period  of  3  months  from  December  2009  to  February  201 0.  Trauma 
cases  that  occurred  between  9  am  and  9  pm  (Monday-Thursday) 
were  observed.  The  researcher  logged  observations  using  an  auto¬ 
mated  data  collection  tool  run  on  laptop  using  Microsoft  Windows® 
XP  operating  system.  The  software  automatically  time-stamps  all 
observations  entered  into  the  system.  In  this  manner,  observations 
were  gathered  unobtrusively.  Clarifications  about  the  events  that 
occurred  were  obtained  from  clinicians  between  Trauma  events. 

Within  the  time  period  specified,  a  total  of  10  Trauma  cases 
were  observed  with  seven  attending  Trauma  surgeons  (experts) 
and  seven  Trauma  residents  at  the  PGY1  (novices)  and  PGY3  (inter¬ 
mediate  expertise)  level,  each.  The  Trauma  cases  were  of  two  types 
-  Trauma  A  and  Trauma  B.  At  the  Trauma  center  in  Banner  Good 
Samaritan  Health  System,  Trauma  A  refers  to  high  criticality  cases 
that  require  the  presence  of  an  anesthesiologist,  while  Trauma  B 
cases  are  those  cases  that  are  classified  as  low  criticality.  Out  of 
the  10  cases  observed,  eight  cases  were  classified  as  Trauma  B 
and  two  as  Trauma  A  cases.  The  ATLS  protocol  for  Initial  Assess¬ 
ment  and  Management  was  utilized  to  assess  these  cases  for  devi¬ 
ations.  Irrespective  of  the  type  of  the  cases,  all  steps  of  the  Initial 
Assessment  and  Management  Protocol  are  required  to  be  followed 
by  the  core  Trauma  team.  This  allows  for  a  valid  comparison  be¬ 
tween  the  10  Trauma  cases. 

The  analysis  of  the  data  was  performed  by  researchers  in  collab¬ 
oration  with  an  expert  Trauma  clinician  (an  attending).  Deviations 
identified  (through  consensus)  are  classified  as  errors,  innovations 
or  procedural  deviations  based  on  the  classification  methodology 
described  in  Section  2.  The  data  set  was  then  analyzed  using  statis¬ 
tical  means  and  interpreted  to  answer  the  questions  outlined  in  Sec¬ 
tion  2.  We  employed  independent  group  t- test  to  find  differences 
between  number  and  types  deviations  in  Trauma  A  and  Trauma  B 
cases.  Ap-value  ofp  <  0.05  was  accepted  as  statistically  significant. 

4.  Results 

4A.  Question  1  -  how  often  do  Trauma  team  members  deviate  from 
the  Advanced  Trauma  Life  Support  protocol? 

The  results  are  presented  as  mean  (p)  ±  standard  deviation  (cr). 
Fig.  2  depicts  the  mean  deviations  that  occurred  in  the  10  Trauma 
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Fig.  2.  Mean  deviations  per  Trauma  case. 
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Fig.  3.  Distribution  of  deviation  and  errors  in  two  Trauma  settings. 

cases  for:  (i)  Trauma  A  and  Trauma  B  (9.1  ±2.14),  (ii)  Trauma  A 
(14  ±  1.41),  and  (iii)  Trauma  B  cases  (7.5  ±  2.79). 

The  mean  number  of  deviations  in  Trauma  A  cases  were  higher 
compared  to  the  mean  deviations  in  Trauma  B  cases.  Typically, 
Trauma  A  cases  involve  unstable  and  unpredictable  patients.  Con¬ 
sequently,  the  Trauma  team  makes  relatively  a  larger  number  of 
deviations  to  adapt  to  the  dynamic  situation  at  hand. 

4.2.  Question  2  -  when  clinicians  deviate,  what  are  the  types  of 
deviations  made? 

Fig.  3  shows  the  distribution  of:  (i)  errors  (Trauma  A :  ju  = 
1.5  ±1.06,  Trauma  B:  p  =  2.63  ±1.1),  (ii)  innovations  (Trauma  A: 
fi  =  0.5  ±  0.35,  Trauma  B:  p  =  0.75  ±  0.7),  (iii)  proactive  procedural 
deviations  (Trauma  A:  p  =  0.5  ±0.35,  Trauma  B:  p  =  0.38  ±  0.37), 
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Fig.  4.  Total  number  of  deviations  as  a  function  of  errors  and  innovation. 
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Fig.  5.  Total  number  of  deviations  as  a  function  of  procedural  deviations. 

and  (iv)  reactive  procedural  deviations  (Trauma  A:  ji  =  1 1 .5  ± 
1.06,  Trauma  B:  ft  =  4.13  ±  1.15). 

From  Fig.  3,  we  can  see  that  errors  make  up  a  small  percentage 
(26.38%)  of  the  total  deviations  in  the  10  Trauma  cases.  This  is  an 
important  result  from  these  observations  as  it  points  to  the  limita¬ 
tions  of  the  current  strategy  of  marking  most  deviations  as  errors 
in  assuring  compliance  to  a  protocol.  The  procedural  deviation 
were  significantly  higher  in  Trauma  A  when  compared  to  Trauma 
B  cases  (p  <  0.05).  The  critical  condition  of  the  patients  in  Trauma 
A  cases  and  the  individual  nature  of  the  problem  cause  the  Trauma 
team  to  deviate  often  in  order  to  manage  the  unique  situation  at 
hand.  Our  analysis  also  showed  that  most  procedural  deviations 
were  reactive  in  nature  in  both  Trauma  A  and  Trauma  B  cases.  This 
can  be  attributed  to  the  dynamic  nature  of  the  critical  care  envi¬ 
ronment.  Clinicians  are  required  to  react  quickly  to  the  changes 
to  ensure  efficient  operation  in  Trauma. 

4.3.  Question  3  -  how  do  these  types  of  deviations  vary  with  the 
experience  (level  and  type)  of  the  members  of  the  clinical  team? 

Figs.  4  and  5  depict  the  total  number  of  errors,  innovations  and 
procedural  deviations  made  by  core  team  members  in  the  10  Trau¬ 
ma  cases  observed. 
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4.3 A.  Errors  and  innovations 

In  this  study,  the  experts  made  no  errors  as  defined  in  our  ana¬ 
lytic  framework.  As  we  consider  care  givers  with  lesser  expertise 
(from  the  3rd  and  4th  year  resident  to  the  1st  and  2nd  year  resi¬ 
dents),  we  saw  a  decline  in  innovation  and  an  exponential  increase 
in  the  number  of  errors,  as  expected.  This  once  again  supports  our 
hypothesis  that  experts’  deviations  are  more  often  innovations 
than  errors,  while  novices’  deviations  lead  most  often  to  errors. 
Trauma  nurses  and  Technicians  show  little  evidence  of  innovation. 
While  this  evidence  cannot  be  attributed  to  a  lack  of  experience,  it 
can  be  hypothesized  that  within  the  confines  of  their  roles  in  inter¬ 
acting  with  a  patient,  there  is  not  much  scope  of  innovation.  Nurses 
and  technicians  are  trained  to  follow  a  strict  protocol  to  support 
the  Trauma  team  and  that  training  may  be  responsible  for  the  ob¬ 
served  patterns. 

4.3.2.  Proactive  and  reactive  deviations 

Fig.  5  provides  a  snapshot  of  distribution  of  procedural  devia¬ 
tions  within  the  Trauma  team.  Banner  Health  System,  being  a 
teaching  hospital,  requires  all  Trauma  cases  to  be  led  by  a  senior 
resident  (PGY3/4)  under  the  supervision  of  an  Attending  clinician, 
or  a  junior  resident  (PGY1/2)  under  the  supervision  of  the  senior 
resident  and  the  attending  clinician.  Trauma  nurses  assist  in  all 
Trauma  cases.  Fig.  5  shows  that  senior  residents  make  the  most 
reactive  procedural  deviations  (as  they  are  performing  bulk  of 
the  protocol),  followed  by  the  Trauma  Nurses.  Junior  residents 
who  generally  assist  but  may  lead  a  few  Trauma  cases  also  made 
a  significant  number  of  procedural  deviations.  These  observations 
show  that  leadership  role  and  associated  tasks  may  be  connected 
to  generating  deviations  to  the  protocol. 

5.  Discussion 

Protocols  and  standards  are  based  on  observations  and  evidence 
gathered  from  practices.  New  information  and  novel  findings  from 
practice  need  to  be  incorporated  into  the  guidelines  and  protocols. 
So  how  do  such  novel  ideas  get  generated  from  practice?  When 
regular  or  standard  patterns  do  not  fit  or  match  the  current  prob¬ 
lem,  possible  alternative  ideas  get  generated.  This  is  the  process  of 
innovation,  and  innovation  is  not  possible  without  deviations.  As 
practitioners  gain  experience  in  the  execution  of  a  task,  their  per¬ 
formance  become  increasingly  smooth  and  efficient.  While  devel¬ 
oping  proficiency  with  attention-demanding  complex  tasks,  some 
component  skills  become  automatic,  so  that  conscious  processing 
can  be  devoted  to  reasoning  and  reflective  thought  with  minimal 
interference  in  the  overall  performance.  A  great  deal  of  experts’ 
knowledge  is  finely  tuned  and  highly  automated  enabling  them 
to  execute  a  set  of  procedures  in  an  efficient  manner.  Yet  they 
can  perform  such  tasks  in  a  highly  adaptive  manner  which  is  sen¬ 
sitive  to  shifting  contexts. 

Our  study  provides  supportive  evidence  for  the  claim  that  devi¬ 
ations  do  occur  in  critical  care  environments  and  not  all  deviations 
are  errors.  Deviations  to  the  protocol  can  be  important  innovations 
and  are  tied  to  complex  decision  making  and  judgment  calls  at  the 
point  of  care.  The  results  are  promising  and  suggest  a  need  for  the 
development  of  ontology  of  deviations  in  Trauma  and  other  critical 
care  environments.  The  recognition  of  deviations  utilizing  such 
ontology  that  classifies  deviations  as  errors,  innovations  and  proce¬ 
dural  deviations  can  significantly  alter  compliance  procedures  and 
provide  an  overall  adaptive  framework  to  modification  of  existing 
protocols.  For  example,  if  deviations  are  consistently  seen  on  a  par¬ 
ticular  step  in  a  protocol,  then  that  step  may  have  to  be  re-ana¬ 
lyzed.  Similarly  if  innovations  are  continuously  seen  and 
replicated  in  multiple  sites,  then  it  could  be  incorporated  into  the 
next  version  of  protocol.  Such  an  ontology  could  allow  for  a  scien¬ 
tific  framework  for  modification  of  protocols  and  enable  protocol 


developers  to  leverage  a  data  driven  approach  to  modifications. 
Currently  available  tools  such  as  checklists,  protocols  need  to  allow 
for  note  takers  to  mark  and  document  deviations,  errors  and 
innovation. 

Such  ontology  could  also  enable  the  development  of  simulators 
driven  by  real-world  data  that  provide  training  to  maximize  inno¬ 
vation  and  minimize  error  occurrence.  Such  an  educational  tool 
will  be  critical  in  developing  decision  making  skills  of  residents 
and  care  givers.  It  would  allow  for  a  comprehensive  evaluation  of 
the  skills  of  the  caregivers  as  well  as  a  means  to  train  teams  for 
not  only  adherence  to  a  protocol  but  enabling  recognition  of  cir¬ 
cumstances  where  innovation  is  needed. 

One  limitation  of  this  study  is  the  number  of  Trauma  cases  stud¬ 
ied.  With  our  current  methods,  it  is  a  challenging  task  to  study 
more  cases,  primarily  because  Trauma  is  an  unpredictable  environ¬ 
ment  and  it  is  hard  to  anticipate  occurrence  of  events,  and  a  great 
deal  of  data  have  to  be  collected  for  analysis.  We  have  recently 
developed  a  system  for  capturing  events  automatically  using 
radio-frequency  identification  (RFID)  systems  [20].  Clinicians  in  a 
Trauma  team  wear  electronic  RFID  tags  that  automatically  track 
their  movement  and  activities.  We  will  leverage  this  system  to 
gather  data  from  Trauma  centers  in  an  automated  manner.  In  addi¬ 
tion  to  tracking  events,  the  system  allows  for  playing  back  events 
in  the  virtual  world.  This  can  enable  more  efficient  data  annotation 
and  collection. 

6.  Conclusion 

Clinicians  deviate  from  protocols  when  managing  patients.  Our 
study  shows  that  clinical  teams  in  critical  care  environments  make 
significant  number  of  deviations  per  case  and  not  all  deviations  are 
errors.  The  study  of  these  deviations  can  provide  new  insight  into 
how  teams  operate  in  complex  environments  and  what  distin¬ 
guishes  experts  from  novices.  The  results  are  in  coherence  with 
existing  literature  on  exploring  cognitive  basis  of  clinical  expertise. 
We  can  hypothesize  that  existence  of  retrieval  structures  in  ex¬ 
perts  and  top  down  information  processing  allows  for  time  critical 
thinking  that  supports  innovation  in  experts.  This  is  supplemented 
by  the  information  filtering  that  the  retrieval  structures  support. 
On  the  other  hand,  novices  are  driven  by  bottom  up  reasoning 
mechanisms  and  without  retrieval  structures  and  filtering  are 
overwhelmed  by  the  data  and  often  make  errors.  While  only  fur¬ 
ther  experimentation  can  investigate  this  hypothesis,  our  observa¬ 
tions  clearly  point  to  the  plausibility  of  such  mechanisms. 

An  analysis  of  deviations  can  enable  building  models  of 
expertise  and  workflow  that  can  be  then  used  to  design  the  next 
generation  of  effective  interventions.  Interventions  could  be  stan¬ 
dardized  communication  tools,  to  information  technology  that 
supports  innovations  by  effective  presentation  of  information 
and  cognitive  decision  support  to  educational  efforts  such  as  sim¬ 
ulations.  Simulations  offer  an  exciting  means  of  teaching  clinical 
care  givers  to  learn  how  to  effectively  innovate  in  complex  envi¬ 
ronments.  Accreditation  Council  of  Graduate  Medical  Education 
recognizes  simulation  as  an  effective  means  of  promoting  critical 
thinking,  professionalism  and  clinical  knowledge  [21].  It  is  gener¬ 
ally  seen  only  as  an  effective  means  of  promoting  standardization 
and  adherence  to  a  protocol  [19].  This  study  however,  shows  that 
simulation  should  be  used  for  teaching  clinical  care  givers  the 
nuances  of  errors  and  innovations.  Simulation  offers  a  safe  envi¬ 
ronment  to  achieve  such  goals.  We  hope  to  develop  such  simula¬ 
tions  that  are  not  just  a  means  of  achieving  standardization  but 
also  help  develop  certain  knowledge  structure  fairly  quickly 
through  practice  that  would  make  any  deviations  safer.  The  data 
presented  in  this  paper  suggests  that  there  is  a  strong  link  between 
innovations,  errors  and  expertise.  Expert  care  givers  deviate  from 
the  protocol  almost  as  often  as  novices  but  make  significantly  more 
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innovations.  This  seems  to  suggest  that  expert  have  a  strong  men¬ 
tal  model  of  how  and  when  to  innovate  and  can  employ  their 
knowledge  and  application  abilities  to  innovate  on  the  fly.  Such 
innovations  and  recognizing  them  should  be  an  important  part  of 
clinical  practice  as  it  helps  is  redesigning  protocols  and  procedures. 

Future  studies  will  explore  in  detail  the  underlying  mechanisms 
of  expertise  and  innovations  in  Trauma.  The  methodologies  de¬ 
scribed  by  Arocha  et  al.  [22]  will  be  employed  for  these  studies. 
Specifically,  we  will  focus  on  semantic  analysis  as  a  means  of 
studying  the  innovations  process  in  experts  and  novices.  We  ex¬ 
pect  that  semantic  analysis  will  yield  important  insights  into 
how  information  is  assimilated  and  processed  by  clinical  care  giv¬ 
ers.  This  would  be  crucial  in  understanding  how  to  develop  novel 
protocols  and  standards.  For  example,  given  the  seriality  of  infor¬ 
mation  as  it  passes  from  working  memory  to  long  term  memory 
[23],  one  may  include  markers  within  the  case  description  that 
may  invoke  the  correct  knowledge  structures  in  the  long  term 
memory  that  support  creativity.  Continuation  of  this  research  will 
enable  us  to  test  such  interventions  (including  simulations  men¬ 
tioned  above)  and  evaluate  them. 

We  also  plan  to  apply  the  same  methodologies  to  study  team  cre¬ 
ativity,  innovations  in  Trauma  environments.  An  important  element 
of  clinical  care  today  is  teamwork  and  often  has  teamwork  can  over¬ 
shadow  individual  innovations.  Teamwork  involves  professional¬ 
ism,  communication  and  situation  awareness  and  innovations 
need  to  be  catalyzed  by  a  supportive  infrastructure  within  teams. 
We  intend  to  investigate  mechanisms  of  creativity  and  innovations 
in  complex  Trauma  environments  at  a  team  level  to  facilitate  devel¬ 
opment  of  standards,  protocols  and  communication  tools. 

Acknowledgments 

This  work  was  support  in  part  by  Grant  from  the  James  S. 
McDonnell  Foundation  (JSMF  220020152).  We  would  like  to  thank 
the  Trauma  team  at  Banner  Good  Samaritan  for  their  support  and 
cooperation  in  this  research  study. 

References 

[1]  Plsek  PE,  Greenhalgh  T.  Complexity  science:  the  challenge  of  complexity  in 
health  care.  BMJ  2001;323:625-8. 


[2]  Gilmour  D,  Woodwards  S.  The  surgical  safety  checklist:  legacy  and  next  step.  J 
Preoperat  Pract  2010;20:78. 

[3]  Duane  T,  Brown  H,  Borchers  C,  Wolfe  L,  Malhotra  A,  Aboutanos  M,  et  al.  A 
central  venous  line  protocol  decreases  bloodstream  infections  and  length  of 
stay  in  a  trauma  intensive  care  unit  population.  Ann  Surg  2009;75:1166-70. 

[4]  Semel  ME,  Resch  S,  Haynes  AB,  Funk  LM,  Bader  A,  Berry  WR,  et  al.  Health 
Affairs  2010;29:1593-9. 

[5]  The  American  Board  of  Surgery  (ABS).  ABS  to  require  ACLS,  ATLS  and  FLS  for 
general  surgery  certification;  2008. 

[6]  Mayo  Clinic  Awarded  Major  Health  Information  Technology  Grant.  <http:// 
www.mayoclinic.org/news2010-rst/5728.html>. 

[7]  Donchin  Y,  Gopher  D,  Olin  M,  Badihi  Y,  Biesky  M,  Sprung  CL,  et  al.  A  look  into 
the  nature  and  causes  of  human  errors  in  the  intensive  care  unit.  Qual  Saf 
Health  Care  2003;12:143-7. 

[8]  Weingart  SN,  Wilson  RM,  Gibberd  RW,  Harrison  B.  Epidemiology  of  medical 
error.  BMJ  2000;320:774-7. 

[9]  Rothschild  JM,  Landrigan  CP,  Cronin  JW,  Kaushal  R,  Lockley  SW,  Burdick  E,  et  al. 
The  critical  care  safety  study:  the  incidence  and  nature  of  adverse  events  and 
serious  medical  errors  in  intensive  care*.  Crit  Care  Med  2005;33:1694-700. 

[10]  Langewiesche  W.  Fly  by  wire:  the  geese,  the  glide,  the  miracle  on  the 
hudson.  New  York:  Farrar,  Straus  and  Giroux;  2009. 

[11]  Patel  VL,  Groen  GJ,  Arocha  JF.  Medical  expertise  as  a  function  of  task  difficulty. 
Mem  Cogn  1990;18:394-406. 

[12]  Groen  G,  Patel  V.  Medical  problem-solving:  some  questionable  assumptions. 
Med  Educ  1985;19:95-100. 

[13]  Patel  V,  Arocha  J,  Kaufman  D.  Diagnostic  reasoning  and  medical  expertise. 
Psychol  Learn  Motiv  1994;31:187-252. 

[14]  Ericsson  K,  Patel  V,  Kintsch  W.  How  experts’  adaptations  to  representative  task 
demands  account  for  the  expertise  effect  in  memory  recall:  comment  on 
Vicente  and  Wang  (1998).  Psychol  Rev  2000;1:578-92. 

[15]  Joseph  G,  Patel  V.  Domain  knowledge  and  hypothesis  generation  in  diagnostic 
reasoning.  Med  Decis  Mak  1990;10:31-46. 

[16]  Finke  R,  Ward  T,  Smith  S.  Creative  cognition:  theory,  research  and 
applications.  Cambridge,  MA:  MIT  Press;  1996. 

[17]  American  College  of  Surgeons.  Advanced  trauma  life  support  for  doctors.  7th 
ed.  Chicago,  IL:  American  College  of  Surgeons;  2004. 

[18]  Collicott  PE,  Hughes  I.  Training  in  advanced  trauma  life  support.  JAMA 
1980;243:1156-9. 

[19]  Kahol  K,  Leyba  MJ,  Deka  M,  Deka  V,  Mayes  S,  Smith  M,  et  al.  Effect  of  fatigue  on 
psychomotor  and  cognitive  skills.  Am  J  Surg  2008;195:195-204. 

[20]  Vankipuram  M,  Kahol  K,  Cohen  T,  Patel  V.  Toward  automated  workflow 
analysis  and  visualization  in  clinical  environments.  J  Biomed  Informatics; 
2011;44:432-40. 

[21]  ACGME.  ACGME  competencies:  suggested  best  methods  for  evaluation 
[document  on  the  Internet].  ACGME;  September  2000.  <http://www.acgme. 
org/Outcome/assess/ToolTable.pdf>  [cited  12.12.07] 

[22]  Arocha  JF,  Wang  D,  Patel  VL.  Identifying  reasoning  strategies  in  medical 
decision  making:  a  methodological  guide.  J  Biomed  Informatics  2005;38: 
154-71. 

[23]  Groen  G,  Patel  V.  The  relationship  between  comprehension  and  reasoning  in 
medical  expertise.  In:  Chi  M,  Glaser  R,  Farr  M,  editors.  The  nature  of 
expertise.  Hillsdale,  NJ:  Erlbaum;  1988.  p.  287-310. 


Collaborative,  Time-Critical,  Multi-Sensory  Training  in 
Virtual  Worlds  with  Persuasive  Elements 


P  Khanal,  S  Parab,  K  Kahol,  Marshall  Smith 
Submitted  to  Computer  Human  Interaction  (CHI),  2011 


ABSTRACT 

Collaborative  Virtual  Environments  (CVEs),  also  known  as 
virtual  worlds,  are  capable  of  making  remote  team  training 
ubiquitous.  However,  the  current  generation  of  the  virtual 
worlds  does  not  support  usage  of  multiple  sensors.  In  this 
study,  we  present  the  design  of  an  interactive  collaborative 
team  training  system  in  a  virtual  world  that  integrates  a 
multisensory  device.  We  consider  a  haptic  joystick  as  a 
multi-sensory  device  and  Advanced  Cardiac  Life  Support 
(ACLS)  as  a  case  study  for  time-critical  team-based 
activity.  To  motivate  users  during  training,  various 
persuasive  components  as  well  as  formative  feedback 
components  are  also  included  in  the  system.  After  the 
training  in  the  virtual  world,  the  participants  were  tested  in 
a  real  training/testing  facility  in  front  of  expert  evaluators. 
Comparison  of  the  performance  of  the  groups  that  were 
trained  in  this  system  to  the  groups  that  were  given 
conventional  didactic  training  in  classroom  settings  shows 
that  the  former  group  performs  better  than  the  latter. 
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INTRODUCTION 

Teamwork  is  considered  to  be  one  of  the  crucial  aspects  to 
attain  success  in  several  domains  such  as  healthcare, 
military  and  aviation  (Crew  Resource  Management)  [8]. 
Effective  and  efficient  team  performance  requires  every 
team  member  to  excel  in  their  individual  roles  and  co¬ 
ordinate  their  actions  with  other  team  members.  Although 
training  the  whole  team  together  has  been  observed  to  be 
more  effective  in  improving  team  performance,  in  clinical 
environments  training  a  member  individually  is  usually 
given  more  importance  [5].  There  are  various  reasons 
behind  this  discrepancy.  One  reason  is  that  it  is  often 
difficult  to  set  up  training  sessions  according  to  each 
individual’s  schedule;  a  team  may  consist  of  members  from 
disparate  locations.  In  such  a  scenario,  individual  training  is 
easier  to  conduct  and  requires  less  time,  cost  and  co¬ 
ordination.  The  time  criticality  and  various  skills  related 
with  a  team-based  activity  make  it  more  difficult  to  provide 
team  training  as  the  team  coordination  and  proper 
communication  between  the  team  members  play  a 
significant  role. 

One  way  to  train  the  team  members  simultaneously  is  via 
the  use  of  virtual  worlds,  also  known  as  collaborative 
virtual  environments  (CVE).  CVEs  are  very  useful  in 
developing,  planning,  and  organizing  collaborative  team 
training  systems  [3].  Virtual  worlds  provide  an  environment 
where  users  can  simultaneously  log  in  from  different 
locations,  and  perform  individual  as  well  as  collaborative 
tasks  with  proper  communication  between  all  team 
members.  For  these  reasons,  virtual  worlds  prove  very 
useful  in  enhancing  cognitive  skills  of  users. 

Apart  from  the  regular  input  and  output  devices  (keyboard, 
mouse,  microphone,  speakers,  and  visual  display),  current 
generation  of  virtual  world  based  training  simulators  does 
not  facilitate  the  use  of  multiple  sensors.  Complex  team 
training  scenarios  often  require  psychomotor  skills  in 
addition  to  simple  cognitive  skills.  Therefore,  the  use  of 
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force  feedback  sensors,  more  commonly  known  as  haptic 
devices,  are  important  in  the  development  of  next 
generation  virtual  world  based  training  simulators. 

It  is  known  that  the  use  of  persuasive  components  help  in 
the  improvement  of  the  performance  of  a  team  [10]. 
Considering  this  knowledge,  we  have  also  added  persuasive 
elements  in  the  team  training  system.  In  medical 
training/education,  persuasion  is  one  of  the  most  important 
factors  that  can  affect  the  performance  of  trainees/students. 
Use  of  meaningful  persuasive  components  such  as  rewards, 
realism  and  social  presence  enhances  the  learning  where  as 
bad  design  of  persuasive  components  hinders  it.  Our  system 
is  designed  in  such  a  way  that  participants  are  provided 
with  various  performance  feedback  parameters  during  the 
training  sessions. 

In  this  study,  we  try  to  study  the  effect  of  virtual  world 
based  team  training  for  time  critical  scenarios  that  require 
both  cognitive  as  well  as  psychomotor  skills  to  solve  the 
problem.  We  also  study  the  effect  of  persuasive  elements 
and  feedback  components  used  in  the  training  system,  and 
use  the  Advanced  Cardiac  Life  Support  (ACLS)  procedure 
as  a  case  study.  ACLS  is  a  time  critical  team  based  activity 
that  refers  to  clinical  interventions  intended  to  treat  life 
threating  medical  emergencies  such  as  cardiac  arrests  and 
respiratory  failures.  ACLS  team  comprises  of  four  or  more 
members  to  take  care  of  a  patient  with  such  emergencies 
[11].  Each  member  in  a  team  is  required  to  have  several 
cognitive  skills  such  as  understanding  emergency 
pharmacology,  managing  patient’s  airway,  and  interpreting 
electrocardiograms.  It  also  requires  psychomotor  skills  like 
Cardiopulmonary  Resuscitation  (CPR),  which  is  an 
artificial  chest  compression  technique  that  requires 
maintaining  constant  rate  and  depth  during  compression 
and  recoil  of  the  chest  of  the  patient.  The  ACLS  team  must 
perform  the  cognitive  and  psychomotor  tasks  within  5 
minutes  with  proper  coordination  between  the  team 
members  to  save  a  patient.  However,  we  will  focus  only  on 
the  procedural  training  of  the  ACLS  procedure,  i.e.,  training 
participants  on  step-by-step  tasks  till  the  completion  of  the 
procedure.  This  does  not  involve  the  clinical  skills  like 
injecting  needles  and/or  putting  oxygen-bag  properly  on 
patient’s  face. 

We  develop  the  training  simulator  and  evaluate  its 
effectiveness  in  enhancing  medical  education  by  testing  its 
use  on  a  team  of  test  subjects.  We  also  design  evaluation 
technique  to  evaluate  the  team  performance  during  the 
ACLS  training,  both  in  virtual  world  and  actual  training 
scenarios.  Our  evaluation  results  show  that  the  teams 
provided  with  our  virtual  ACLS  training  system  perform 
much  better  than  other  teams  that  are  not  provided  with  the 
training. 

The  remainder  of  this  paper  is  organized  as  follows:  Some 
of  the  previous  studies  on  team  training,  persuasive 
components  and  medical  education  in  virtual  world  are 
explained  in  “Related  Work”  section.  “Design 


Methodology”  describes  various  design  components  of  the 
system.  Experimental  design  and  results  are  explained  in 
“Experiment  and  Results”  section.  We  discuss  the  results  in 
the  Discussion  section  and  finally  conclude  our  paper  in  the 
“Conclusion”  section. 

RELATED  WORK 

Callaghan  et.  al.  used  Second  Life  to  create  a  virtual 
learning  environment  for  engineering  education  [3].  They 
demonstrated  various  interactive  simulations  that  are  part  of 
engineering  education.  Apart  from  the  simulations,  a  virtual 
lecture  theater  is  also  present  in  the  virtual  world  that 
contains  an  interactive  slideshow  viewer,  and  a  media 
center  for  streaming  video  content  and  message  centers  for 
feedback.  After  simulation  training,  the  participants  are 
asked  to  take  a  quiz;  if  they  answer  incorrectly  they  have  to 
run  the  simulation  again  and  answer  the  questions  correctly. 
The  system  provides  very  less  interaction  mechanisms  to 
the  users.  The  study  lacks  the  assessment  and  the  evaluation 
of  participants  and  they  mention  that  these  shortcomings 
will  be  their  main  focus  in  the  future. 

Weicha  et.  al.  [2]  explored  the  potential  of  a  virtual  world, 
Second  Life  (SL),  as  a  teaching  tool  for  continuing  medical 
education  (CME).  In  their  study,  10  participant  physicians 
were  made  to  select  and  adjust  insulin  level  for  patients 
with  type-2  diabetes.  Participants  had  to  listen  to  an 
instructional  40-minute  insulin  therapy  talk.  The  study  was 
designed  such  that  the  participants  had  to  take  a 
questionnaire  both  before  and  after  their  training.  The  study 
noted  significant  improvement  in  the  participants  and 
revealed  that  virtual  worlds  are  very  helpful  for  continuing 
medical  education.  However,  this  system  was  only 
replicating  the  classroom  training  in  the  virtual  world  rather 
than  providing  an  interactive  simulated  environment  for  the 
participants  to  promote  do-then-leam  learning. 

Boulos  et.  al.  described  the  potential  use  of  Second  Life  in 
medical  and  health  education  [6].  The  authors  provided  two 
scenarios,  ‘Virtual  Neurological  Education  Centre’  (VNEC 
2006)  and  ‘Healthlnfo  Island’  (Info  2005).  The  former 
demonstrates  a  scenario  where  users  are  exposed  to  most 
common  neurological  disability  symptoms.  Apart  from  the 
symptoms,  they  are  also  provided  with  related  information, 
events,  and  facilities  in  the  Second  Life.  The  latter  involves 
providing  training  programs  for  virtual  communities.  It  also 
intends  to  provide  support  to  Second  Life  residents  by 
giving  them  opportunities  to  participate  in  different  medical 
groups  dealing  with  stroke  support  and  cerebral  palsy. 

The  research  study  performed  by  Chodos  et.  al.  [1]  focuses 
on  the  development  of  a  virtual  environment  to  enhance 
communication  skills  for  health  science  education.  They 
talk  about  two  case  studies.  The  first  one  is  the 
development  of  EMT/ER  training  simulation,  which 
delivers  an  environment  to  train  EMT/ER  personnel  on 
taking  care  of  an  accident  victim  before  taking  him  to  a 
hospital.  This  case  also  focuses  on  the  exchange  of  patient 
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information  between  EMT  and  ER  personnel.  The  second 
case  is  designed  to  teach  various  competencies  to  students 
such  as  rehabilitation  medicine,  nutrition,  and  physical 
education.  For  the  second  case,  the  authors  design  a 
simulation  in  order  to  increase  communication  between  the 
students  to  develop  a  home-care  plan  for  the  elderly  patient. 
Based  on  the  case  studies,  they  discuss  the  expectations  of 
students  towards  virtual  world  based  learning  and  the 
quality  of  learning. 

There  have  been  some  attempts  at  developing  team  training 
for  Advanced  Cardiac  Life  Support  (ACLS).  Simulation 
training  has  proven  to  be  one  of  the  best  solutions  available 
to  date.  According  to  Wayne  et.  al.  [7],  simulator  training 
has  shown  significant  performance  improvement  in  a  team 
of  physicians  while  performing  ACLS. 

Burleson  et.  al.  talk  about  potential  of  affective  agents 
(affective  learning  companions)  in  influencing  perseverance 
in  the  face  of  failure  [9].  They  show  that  subjects  can  be 
motivated  to  learn  through  failures  using  affective  agents. 
In  this  study,  they  propose  a  system  where  the  subjects  have 
to  solve  the  Towers  of  Hanoi  puzzle.  The  affective  agents 
offer  help  if  the  user  is  facing  failure  or  the  user  is  stuck  at 
some  point,  otherwise  they  allow  the  user  to  explore  the 
puzzle  by  themselves.  They  state  that  such  motivation 
positively  influences  users  without  hampering  their 
originality. 


Design  Components 

Figure  (1)  shows  the  design  of  the  system  from  the 
perspective  of  a  user  who  is  performing  the  CPR  action. 
The  system  can  be  divided  into  four  components:  CVE 
component,  haptic  component,  voice  component,  and 
database  component. 
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Figure  1.  System  Design. 


Even  though  team  training  has  evidently  shown  significant 
performance  improvements  in  participants;  individual 
training  is  given  more  preference.  The  reasons  being  there 
are  increased  difficulties  in  setting  up  training  sessions 
according  to  each  individual’s  schedule  with  everyone  from 
disparate  locations.  Moreover,  inaccurate  user  validation 
schemes  and  inadequate  feedback  provided  by  the  system 
limit  the  user’s  ability  to  learn. 

DESIGN  METHODOLOGY 
Design  Cases 

As  mentioned  earlier,  we  are  considering  using  the  ACLS 
procedure  for  the  case  study  of  time-critical,  multi-sensory, 
and  collaborative  training  in  virtual  world.  There  are 
various  cases  that  need  to  be  considered  while  saving  a 
patient  with  cardio-respiratory  failures.  Based  on  the 
rhythm  of  the  heart,  which  can  be  seen  on 
electrocardiogram  (EKG),  these  cases  can  be  divided  into 
“shockable”  and  “non-shockable”  rhythms.  In  this  study, 
we  consider  two  different  generic  cases:  Ventricular 
fibrillation  (Vfib)  to  represent  shockable  rhythms;  and 
Pulseless  Electric  Activity  (PEA)  for  non-shockable 
rhythms.  Each  of  these  cases  has  its  own  causes  and  cures. 
The  first  thing  that  the  ACLS  team  has  to  do  is  check  the 
pulse  and  if  there  is  no  pulse,  start  the  CPR  and  provide 
artificial  respiration  as  soon  as  possible.  The  team  has  to 
then  identify  the  case,  either  as  shockable  or  as  non- 
shockable,  and  start  the  special  procedures  before  five 
minutes  in  order  to  save  the  patient. 


Visual  Component 

Visual  component  includes  all  visual  objects,  avatars,  and 
animation  sequences  in  the  CVE.  We  used  Active  Worlds 
(AW)  [13]  as  our  CVE.  A  virtual  hospital  was  created  by 
using  as  the  model  the  original  floor  plans  of  a  real  ACLS 
training/testing  center  in  a  hospital.  We  then  created  various 
objects  and  custom  avatars  of  doctors  and  nurses  using 
Maya®,  and  3D  Studio  Max®.  The  avatars  are  the  visual 
representations  of  the  users  in  the  virtual  environment. 
Multiple  users  can  log  into  the  AW  simultaneously.  They 
can  select  their  own  avatars,  and  navigate  around  the  virtual 
hospital.  The  avatars  can  perform  various  gestures  like 
flying,  running,  chest  compression,  and  checking  the  pulse 
of  the  patient.  Changes  in  a  virtual  scene  are  visible  to  all 
users  who  are  available  to  the  scene.  The  objects  are 
converted  into  Renderware®  object  format,  which  is  the 
native  format  for  AW.  Various  animation  sequences  are 
then  applied  to  the  objects  that  include  showing/hiding  and 
moving  objects. 

Haptic  Component 

The  haptic  component  is  responsible  for  measurement  of 
the  CPR  rate  during  training.  We  used  the  Novint  Falcon® 
haptic  joystick  [14]  for  the  system.  The  major  objectives  of 
the  haptic  component  are:  1)  interaction  with  the  haptic 
device,  and  providing  of  proper  force  feedback  to  the  user 
who  is  performing  CPR  using  the  device;  and  2)  sending 
responses  from  the  device  to  the  AW  and  triggering  various 
action  events  in  the  active  worlds.  As  the  force  resolution  of 
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the  haptic  device  is  less  than  required,  we  also  attach  a 
spring  on  the  head  of  the  joystick  so  that  it  can  provide 
realistic  hardness  during  chest  compression.  When  the  user 
performs  the  CPR  action  with  the  haptic  device,  the  device 
provides  a  force  feedback  to  the  user  only.  This  triggers 
his/her  avatar  to  perform  chest  compression  gesture  in  the 
AW,  which  is 


Figure  2.  Virtual  ACLS  training  room  with  required 
objects. 


visible  to  all  users  who  are  logged  in  to  the  scene.  At  the 
same  time,  the  rate  of  the  compression  is  also  recorded. 

Voice  Component 

The  voice  component  consists  of  Skype  [12],  which  runs 
separately  from  the  main  components.  We  choose  Skype 
because  it  provides  good  quality  voice  communication 
between  multiple  parties. 

Database  Component 

The  database  component  is  used  to  record  the  task 
performed  by  each  user  during  a  session.  We  use  MySQL 
server,  which  is  setup  in  a  different  location.  The  users, 
tasks  performed,  relative  time  of  tasks  performed,  and  the 
scores  are  stored  in  the  database.  The  main  purpose  of  using 
the  database  is  to  monitor  the  performance  of  the  users  in 
the  subsequent  virtual  training  sessions;  and  also  for  inter¬ 
group  comparison  of  performance. 

Interaction  design 

There  are  two  kinds  of  interactions  provided  in  the  system. 
The  first  one  is  the  interaction  with  the  haptic  device,  as 
explained  in  the  ‘Haptic  component’  sub-section.  The 
second  type  of  interaction  is  via  mouse-clicks.  The  virtual 
objects  in  the  AW  are  click  enabled  and  corresponding 
events  (or  tasks)  are  triggered  on  each  click.  All  the 
triggered  events  are  stored  in  the  remote  database  server. 


from  realistic  objects  and  environment,  we  also  add  various 
kinds  of  persuasive  components  in  the  system  that  helps  in 
motivating  users  when  performing  their  tasks  in  the  virtual 
environment.  For  instance,  we  use  scores,  instructions,  and 
alerts.  A  ticking  clock  is  placed  on  the  virtual  wall  to  help 
the  team  track  time.  As  persuasion  is  one  of  the  most 
important  factors  that  can  affect  the  performance  of  trainees 
in  medical  education,  we  attempt  to  implement  some  of  the 
concepts  that  Fogg  defined  in  [10].  The  first  concept  that 
we  implement  is  “ Tunneling ”.  According  to  Fogg, 
tunneling  is  a  technique  that  allows  users  to  reach  a  goal  by 
stepwise  instructions.  In  our  training  system,  step-by-step 
instructions  are  also  made  available  in  the  training  mode. 
However,  in  the  testing  mode,  these  instructions  are  not 
available  to  the  users.  Another  concept  defined  by  Fogg  and 
we  implement  in  our  system  is  “Tailoring” ,  in  which  only 
the  relevant  information  to  the  individuals  are  shown.  The 
example  of  tailoring  will  be  showing  decision  nodes  (eg. 
Vfib,  PEA)  and  once  a  case  is  followed,  the  participants  do 
not  need  to  know  information  regarding  other  nodes. 
Another  concept  that  we  implement  in  this  study  is 
“Suggestion  technology ”,  which  is  to  provide  users  with 
timely  reminders  to  perform  certain  tasks.  In  this  system, 
we  provide  users  with  a  virtual  character  that  shows  alerts 
as  well  as  instructions  during  the  training.  Based  on  the 
performance  of  the  team,  the  character  changes  his  facial 
expression;  smiley  when  the  team  is  performing  well,  and  a 
frown  when  it  is  not  performing  well.  This  motivates  users 
to  change  their  behavior  during  the  training  session.  The 
final  concept  that  we  implement  in  our  system  is  “ Self¬ 
monitoring ”.  Self-monitoring  is  a  way  where  users  can 
visualize  their  performance.  In  our  system,  we  provide 
recently  performed  task  as  well  as  corresponding  score  (if 
any)  during  the  training.  Because  of  this,  the  users  are 
always  aware  of  what  they  have  done  and  what  they  have  to 
do  next.  A  team  scores  if  it  can  perform  the  correct  tasks 
within  a  specified  time,  which  is  very  important  for  self- 
evaluation  of  the  performance.  Figure  (3)  shows  various 
persuasive  components  implemented  in  the  training  system. 


PLEASE 
START  CPR 
as  soon  as 
possible. 


The  Oxygen 
bag  is  placed 
on. 


Persuasive  Components 

BJ  Fogg  defined  “Persuasive  elements”  as  the  elements  that 
motivate  users  to  change  their  behavior  or  attitude  [10].  As 
mentioned  by  Conradi  et.  al.  [4],  working  with  realistic 
objects  and  environment,  and  a  feeling  of  being  in  a  group 
motivate  users  to  perform  tasks  in  the  virtual  environment. 
We  add  realistic  models  of  hospitals  and  sounds  within  a 
hospital  that  provide  realistic  ambience  to  the  users.  Apart 


Figure  3.  Various  persuasive  technologies  used  in  the 
system:  (from  left  to  right)  tunneling,  suggestion,  self¬ 
monitoring. 

Performance  Feedback 

The  training  system  uses  two  methods  for  feedback: 
formative  and  summative.  Formative  feedbacks  are  those 
provided  to  the  users  at  various  stages  of  the  training; 
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whereas,  summative  feedback  is  only  provided  at  the  end  of 
the  training. 

Various  alerts  are  provided  based  on  the  performance  of  the 
teams  during  the  training.  The  alerts  pop-up  relative  to  the 
team’s  performance.  If  the  team  is  lagging  behind  in  time 
when  performing  a  certain  task,  alerts  will  pop-up  to  inform 
the  team  that  they  are  lagging.  If  they  are  performing  the 
task  within  the  pre-specified  amount  of  time,  alerts  with 
appreciation  are  displayed  to  motivate  them  to  perform 
better.  Scores  are  also  shown  to  inform  them  that  they  are 
performing  well. 


At  the  end  of  a  training  session,  the  team  will  be  shown  a 
happy  faced  character  if  the  patient  is  saved;  or  a  sad  faced 
character  if  the  patient  dies.  In  addition  to  the  final 
messages,  the  team  members  are  provided  with  a  webpage 
link  that  shows  the  summary  of  their  performance.  The 
summary  page  shows  the  total  points  that  the  team  scored, 
whether  or  not  the  patient  is  saved,  and  the  overall 
performance  of  the  team.  A  summary  table  is  also  shown 
that  lists  the  users’  id,  tasks  performed  by  each  user, 
duration  of  the  performed  task,  and  points  earned  for  the 
performed  task.  There  is  also  an  option  available  to  view 
the  individual  performance  of  a  user.  The  summary  of  the 
overall  training  session  is  retrieved  from  the  database  server 
that  is  setup  in  a  different  machine  in  the  same  network. 
Figure  (4)  (left  and  right)  shows  the  example  of 
implementation  of  summative  feedback  and  formative 
feedback  respectively. 


GOOD  JOB 
GUVS! 

You  have 
managed  to 
save  the 
patient. 

P (ease  be  patient  your 
report  is  being  generated. 


Figure  4.  Summative  feedback  (left)  and  Formative 
feedback  (right)  in  the  virtual  ACLS  training  system. 


EXPERIMENT  AND  RESULT 
Experiment  Design 

There  were  altogether  24  participants  with  no  knowledge  on 
ACLS  prior  to  the  experiment.  We  divided  the  participants 
into  six  groups  of  four  each.  We  asked  each  participant  to 
take  a  quiz,  with  10  multiple  choice  questions  related  to 
ACLS,  prior  to  conducting  the  experiment,  in  order  to 
verify  that  the  participants  are  at  the  same  level  of  expertise 
on  ACLS. 

The  experiment  consisted  of  two  general  groups:  control 
group,  and  experimental  group.  Each  of  the  groups  had  two 
groups  of  four  participants  each.  All  six  groups  were 


provided  with  30  minute  didactic  training  session  on  ACLS 
separately.  The  experimental  group  was  provided  with 
additional  virtual  world  training.  The  experimental  group 
was  further  divided  into  two  more  groups:  procedural  (non¬ 
persuasive)  and  persuasive.  Persuasive  groups  were  trained 
in  the  virtual  world  with  persuasive  elements  like  timely 
alerts  (feedback)  and  scores  enabled  in  it.  Procedural 
groups  were  provided  with  training  in  the  virtual  world  but 
were  provided  with  no  persuasive  elements  as  well  as  no 
formative  feedback  elements. 

In  the  first  phase  of  the  experiment,  the  experimental 
groups  (both  procedural  and  persuasive  groups)  were  first 
introduced  to  the  AW,  and  then  the  virtual  hospital.  After  3- 
5  minutes  of  exploring  in  the  virtual  hospital  they  were 
ready  to  start  the  training  sessions.  The  experimental  groups 
were  provided  with  two  training  sessions  and  a  test  session 
for  each  case  (Vfib  and  PEA)  in  the  virtual  world.  In  the 
training  sessions,  both  the  groups  were  provided  with  step 
by  step  instructions  to  perform  the  set  of  tasks.  The  test 
sessions  for  both  persuasive  and  procedural  groups  did  not 
get  any  alerts,  scores,  or  instructions.  All  the  information 
during  the  training  and  the  testing  sessions  were  stored  in 
the  database  server.  After  training  and  testing  in  the  virtual 
hospital  settings,  the  participants  of  experimental  groups 
were  asked  to  fill  out  isometric  questionnaire  regarding  the 
look  and  feel  of  the  training  system,  and  the  quality  of 
learning  in  the  virtual  environment. 

The  second  phase  of  the  experiment  was  the  testing  of 
transfer  of  skills  from  virtual  world  training  to  actual 
training  room  (Figure  5)  in  a  hospital.  The  next  day  after 
the  first  phase  of  the  experiment,  the  groups  were  taken  to 
the  real  ACLS  training/testing  center  to  test  whether  the 
groups  could  transfer  the  learnt  skill  to  the  actual 
training/testing  room.  At  the  center,  each  group  was 
introduced  to  the  tools,  equipment  and  medications  that 
needed  to  be  used  during  the  testing  session. 


Figure  5.  Participants  in  the  real  training/testing  room 
with  high  fidelity  manikin. 

After  the  exposure  to  the  required 
tools/equipment/medications,  the  groups  were  randomly 
called  into  the  testing  room.  Each  group  was  given  two 
ACLS  cases  chosen  at  random.  The  testing  sessions  were 
organized  in  the  presence  of  certified  ACLS  trainers  who 
were  in-charge  of  evaluation  of  the  teams’  performances. 
As  no  participants  were  familiar  with  the  clinical 
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procedures  like  injecting  syringes  for  medication,  we  were 
more  focused  on  observing  the  procedural  aspects  of  the 
training.  The  procedural  aspects  included  tasks  like 
identifying  rhythms,  identifying  cause,  giving  proper 
medications,  team  communication,  and  the  sequence  of  the 
tasks  performed.  The  two  evaluators  evaluated  the 
performance  of  each  group  and  noted  the  time  taken  by 
each  group  for  performing  each  task.  The  final  evaluation 
for  each  group  was  calculated  using  the  average  values  of 
the  evaluations  by  each  evaluator.  The  time  taken  for  the 
final  evaluation  was  used  to  derive  a  generic  score  that 
would  differentiate  the  performance  of  each  group. 

Results 

The  initial  quiz  prior  to  the  experiment  consisted  of  10 
questions.  The  mean  score  of  the  groups  was  2.87  with 
standard  deviation  0.26.  This  proves  that  all  participants 
had  minimal  but  equal  knowledge  about  ACLS  and  were  at 
almost  the  same  level  at  the  beginning  of  the  experiment. 

After  the  didactic  sessions,  the  groups  were  provided  with 
the  same  quiz  that  they  took  prior  to  attending  the  didactic 
training.  The  average  score  of  each  group  after  the  didactic 
training  was  8.45  +-  0.24.  This  shows  that  didactic  training 
helped  the  participants  to  improve  their  knowledge  on 
ACLS.  It  is  notable  that  their  skills  were  at  almost  the  levels 
even  after  the  didactic  training.  Figure  (6)  shows  the 
performance  of  the  groups  before  and  after  the  didactic 
training  session. 


Didactic  Questionnaire  Results 
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Figure  6.  Performance  of  the  ACLS  groups  before  and 
after  the  didactic  training  session. 

During  the  second  phase  of  the  experiment,  all  groups  were 
trained  and  tested  in  the  AW.  All  the  groups  were  provided 
with  two  training  sessions  and  a  testing  session  for  each 
case.  The  important  information  gathered  in  all  sessions 
was  stored  in  the  database.  For  each  ACLS  case,  average 
score  for  training  and  testing  modes  were  calculated 
separately  for  all  groups.  The  resulting  scores  were  then 
averaged  for  persuasive  and  non-persuasive  groups.  The 
final  percentile  score  is  calculated  by  dividing  the  final 
average  score  by  the  maximum  possible  score  for  the 
corresponding  ACLS  case. 


Figure  (7)  shows  the  performance  of  procedural  and 
persuasive  groups  in  both  training  and  testing  sessions 
within  the  virtual  world.  The  percentile  scores  of  the 
persuasive  and  the  non-persuasive  groups  for  the  Vfib  case 
(during  training  mode)  are  71.3%  and  52.8%  respectively. 
Similarly,  during  test  mode,  the  scores  are  80.6%  and 
65.6%  respectively.  In  case  of  PEA,  during  the  training 
mode,  the  persuasive  group  scored  77.5%,  whereas  the  non¬ 
persuasive  group  scored  67.1%.  During  the  test  mode,  they 
scored  84.2%  and  82.1%  respectively.  These  results  signify 
that  Persuasive  groups  were  better  than  the  non-persuasive 
groups  during  the  virtual  world  training  and  testing. 
However,  in  the  test  case  of  PEA,  there  is  no  significant 
difference  in  the  performance  between  the  two  groups. 

Performance  in  the  virtual  world  (Vfib) 


■  Persuasive 

■  Non-persuasive 


Training  Test 


Performance  in  the  virtual  world  (PEA) 


Figure  7.  Performance  of  persuasive  and  non-persuasive 
groups  in  the  virtual  world  for  Vfib  (above)  and  PEA 
(below). 

As  mentioned  previously,  we  used  a  haptic  joystick  to 
simulate  the  CPR  action.  CPR  rate  and  the  depths  of 
compressions  are  the  two  components  used  to  evaluate  the 
performance  quality  of  the  CPR.  However,  in  this  training 
system,  we  recorded  only  the  rate  during  each  testing  and 
training  session  in  the  virtual  world.  The  comparison  of 
CPR  rate  maintained  by  each  group  (persuasive, 
procedural)  is  shown  in  Figure  (8). 

The  average  CPR  rates  maintained  by  the  persuasive  and 
the  non-persuasive  groups,  in  the  training  mode  for  Vfib 
case,  are  82  and  59  respectively.  In  the  test  mode,  the 
groups  maintained  comparatively  better  rates  at  87  and  75 


6 


respectively.  For  the  PEA  case,  in  the  training  mode,  the 
persuasive  and  the  non-persuasive  groups  maintained  82 
and  69  rates  per  minute;  whereas  in  the  test  mode,  they 
maintained  the  rates  of  98  and  82  respectively.  These 
numbers  suggest  that  the  persuasive  groups  performed 
better  in  maintaining  CPR  rate  than  the  non-persuasive 
group.  However,  it  is  to  be  noticed  that  both  the  groups 
were  not  able  maintain  1 00  compressions  per  minute  during 
CPR  as  suggested  in  the  CPR  guidelines  [11]. 


Figure  8.  CPR  performance  of  the  groups  in  the  virtual 
world:  Vfib  (above)  and  PEA  (below). 

The  final  phase  of  the  experiment  was  conducted  in  the 
actual  training  room  at  the  real  training/testing  center.  The 
training  room  was  fully  equipped  with  all  necessary  tools 
and  equipment  required  for  the  testing.  The  participants  had 
to  perform  CPR  on  the  high-fidelity  programmable 
manikin,  which  was  constantly  monitored  by  an  instructor. 
The  instructor  changed  the  settings  as  the  teams  progressed 
during  the  test  sessions.  Each  team  was  selected  at  random, 
and  presented  with  an  ACLS  case.  The  process  continued 
after  all  the  groups  were  tested  for  both  Vfib  and  PEA 
cases.  Figure  (9)  shows  the  performance  of  each  group  for 
each  case. 


It  can  be  seen  in  Figure  (9)  that  the  performances  of  the 
experimental  groups,  both  persuasive  and  non-persuasive, 
were  much  better  than  those  of  the  control  groups.  The 
persuasive  groups  (PI  and  P2)  outperformed  non¬ 
persuasive  groups  (NP1  and  NP2)  when  they  were 
presented  with  the  Vfib  case.  In  the  case  of  PEA,  one  of  the 
persuasive  groups  (PI)  outperformed  the  non-persuasive 


groups.  The  performance  of  persuasive  group  (P2)  in  PEA 
case  is  worse  than  that  of  both  non-persuasive  groups.  This 
was  due  to  some  technical  problems  related  to  defibrillator 
that  prevented  the  group  from  following  the  procedure 
within  the  pre-specified  time  interval  (marked  by  in 
Figure  9). 


Performance  in  the  real  training/testing  center  (Vfib) 
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Figure  9.  Performance  of  the  groups  in  the  actual 
training  room  at  real  ACLS  training/testing  center:  for 
Vfib  (above)  and  PEA  (below).  [Cl  &  C2:  control  groups 
1  and  2;  PI  &  P2:  persuasive  groups  1  and  2;  NP1  & 
NP2:  non-persuasive  groups  1  and  2J. 

After  the  training  and  testing  in  the  virtual  hospital  settings, 
the  participants  of  experimental  groups  were  asked  to  fill 
out  feedback  questionnaire  regarding  the  look  and  feel  of 
the  training  system,  and  the  quality  of  learning  in  the  virtual 
environment.  Twenty  one  questions  were  given  in  six 
different  categories,  targeted  to  obtain  feedback  about  the 
system  in  order  to  evaluate  its  advantages  and  limitations. 
Each  question  required  the  participant  to  rate  one  of  the 
features  of  the  system  on  a  scale  of  1  to  10,  10  being  the 
best  score.  The  six  different  categories  were  as  follows:  a) 
Ease  of  use  of  the  simulator;  b)  quality  of  force  feedback 
during  CPR  simulation  on  haptic  joystick;  c)  lag 
experienced  in  the  system;  d)  aid  provided  by  persuasive 
elements  during  training  sessions;  e)  improvement  in  ACLS 
skills  due  to  training  using  this  simulator;  f)  overall  rating. 
We  took  the  average  of  the  score  given  for  each  category. 
Figure  (10)  shows  the  qualitative  feedback  provided  by  the 
participants  on  the  simulator.  The  user  feedback  summary 
chart  (Figure  10)  shows  that  the  participants  felt  that  use  of 
simulator  helped  them  to  learn  the  basic  concepts  of  ACLS 
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procedure  in  an  interesting  way.  They  also  felt  that  the 
simulator  was  easier  to  use,  use  of  haptic  device  was 
helpful  for  CPR  training,  and  persuasive  components 
helped  them  to  act  faster  and  in  a  correct  manner.  However, 
they  also  suggested  that  the  simulator  would  be  easier  to 
use  if  the  lag  during  the  training  could  be  reduced.  The  lag 
was  introduced  because  of  the  speed  of  the  internet,  so, 
when  a  participant  clicks  on  some  objects  on  the  scene,  the 
effect  would  be  seen  at  least  one  second  later. 


User  Feedback 


Ease  of  Use  Quality  of  Lag  Effect  of  ACLS  Skills  Overall 
CPR  experienced  persuasive  improved  rating 
simulation  in  the  technology 
System 

Figure  10.  Qualitative  feedback  summary. 

DISCUSSION  AND  FUTURE  WORK 

Ours  is  the  pilot  study  of  a  virtual  world  based  time-critical 
team-training  simulator  with  multi-sensory  device 
integrated  to  it.  We  now  describe  various  design  and 
implementation  factors  that  might  be  helpful  in  designing 
such  simulators  in  the  future. 

Evaluation  of  performance 

The  quality  of  performance  of  a  team  performing  the  ACLS 
procedure  is  highly  dependent  on  the  time  taken  to  perform 
each  individual  task.  To  the  best  of  our  knowledge,  there  is 
no  metric  other  than  time  to  aid  in  quantitative  analysis  of 
the  training.  For  this  reason,  we  designed  a  scoring  system 
based  on  the  time  taken  to  complete  each  task  and  verified 
the  consistency  of  our  method  with  experienced  ACLS 
instructors.  For  Vfib  case,  we  identified  eight  key  tasks, 
which  must  be  performed  within  certain  duration  of  the 
start  of  the  session.  For  each  task,  we  chose  three  “threshold 
duration”  values  and  within  which  the  task  should  be 
performed.  For  each  of  the  “threshold  duration”  values,  we 
also  assigned  corresponding  scores.  For  example,  if  the  task 
is  performed  within  the  “threshold  duration”  with  the  least 
value,  the  team  scores  10  points.  Similarly,  the  score  values 
are  8  and  5  respectively  for  the  tasks  performed  within  the 
next  two  “threshold  durations”.  If  the  team  cannot  finish  the 
task  within  the  “threshold  duration”  with  maximum  value, 
they  earn  0  points  for  the  task.  Similarly  we  divided  the 
PEA  procedure  into  seven  key  tasks  and  used  the  same 
scoring  system.  The  final  percentile  score  is  calculated  as 
follows: 

Total  score  obtained 

Percentile  score  =  100  X  - - - — - 

Total  possible  score 


This  scoring  system  was  consistent  for  the  virtual  world 
training  and  testing,  and  also  the  actual  testing  at  the  real 
training/testing  center. 

Roles  of  different  persuasive  components 

Various  persuasive  components  were  used  in  this  system  to 
motivate  users  to  change  their  behaviors  or  attitude  during 
the  training  sessions.  Each  component  that  was  used  in  this 
system  was  hypothesized  to  have  some  effect  in 
leaming/training. 

Instructions  were  presented  to  the  participants  for  each 
subtask  throughout  the  procedure.  The  intention  of 
displaying  these  instructions  to  the  participants  was  to  allow 
them  to  get  used  to  with  the  simulator.  In  addition,  the 
instructions  would  also  help  the  teams  to  perform  the 
procedure  correctly  and  save  the  patient.  The  visualization 
of  recently  performed  tasks  allows  the  users  to  keep  track  of 
the  current  status  of  the  training  session.  For  the  same 
purpose,  we  also  added  a  timer,  and  scoring  mechanism. 
This  would  motivate  the  participants  to  be  more  focused 
during  the  training.  The  timely  alerts  that  are  displayed 
during  the  game  are  designed  to  suggest  the  participants  on 
how  they  can  improve  their  performance.  The  positive 
alerts  like  “Good  job!  Keep  doing  CPR!”  are  intended  to 
help  them  keep  them  motivated.  On  the  other  hand, 
negative  alerts  like  “You  should  have  started  CPR  by  now!” 
encourage  the  teams  to  perform  remaining  tasks  on  time. 

Effect  of  formative  feedback  on  performance  in  the 
virtual  world 

From  the  experimental  results,  we  can  observe  that  the 
persuasive  groups  performed  better  than  the  non-persuasive 
or  procedural  groups  in  the  virtual  world  training  and 
testing  sessions.  The  major  difference  in  the  experiments 
performed  by  the  two  groups  was  the  use  of  formative 
feedback  in  the  form  of  ‘alerts’. 

Various  timely  alerts  were  presented  to  the  persuasive 
groups  during  the  training  sessions  (not  testing  sessions). 
The  sudden  appearance  of  alerts  triggers  the  participants  to 
initiate  the  tasks  immediately.  Therefore,  in  most  cases,  the 
participants  of  the  persuasive  groups  are  able  to  finish  the 
tasks  on  time,  unlike  the  non-persuasive  ones  who  might 
miss  the  task  deadlines  sometimes.  Apart  from  the  alert,  the 
persuasive  groups  are  also  provided  with  a  wall  clock  to 
track  time  elapsed  since  the  start  of  the  procedure. 

However,  the  formative  feedback  components  were  shown 
to  the  persuasive  groups  during  the  training  sessions  only. 
During  the  test  sessions,  the  training  system  was  designed 
to  work  in  the  exact  manner  for  both  persuasive  and  non¬ 
persuasive  groups.  The  persuasive  groups  performed  better 
than  the  non-persuasive  groups  in  the  test  sessions  as  well. 
From  this  result,  we  can  conclude  that  formative  feedback 
can  help  the  participants  to  retain  the  skills.  More 
interestingly,  both  groups  performed  better  in  the  test 
sessions  than  in  the  training  sessions.  This  improvement 
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might  be  caused  by  the  summative  feedback  provided  at  the 
end  of  each  training  session.  The  summative  feedback  was 
in  the  form  of  performance  evaluation  sheet  where  each 
task  was  listed  along  with  the  time  to  perform  the  task, 
score,  and  the  user  who  performed  the  task.  This  enabled 
the  groups,  as  well  as  each  individual,  to  know  the  tasks 
that  they  could  not  performed  well.  This  form  of  feedback 
might  motivate  them  to  improve  their  performance  in  those 
tasks  in  future  practice  sessions. 

Transfer  of  skills  to  the  real  training  room 

Figure  (9)  shows  the  performance  of  groups  or  teams  in  the 
real  training/testing  center.  The  performances  of  the  control 
groups  were  worse  than  that  of  the  experimental  groups. 
Unlike  the  control  groups,  the  experimental  groups  were 
provided  with  training  in  the  virtual  world.  These  results 
verify  the  fact  that  the  virtual  world  training  can  be  a 
significant  aid  to  the  conventional  didactic  training  for 
ACLS. 

The  performance  comparison  between  the  persuasive 
groups  and  the  non-persuasive  groups  in  the  actual  training 
room  is  not  conclusive.  However,  we  can  see  that  the 
persuasive  group  PI  (refer  to  Figure  8),  outperforms  other 
groups.  The  performance  of  persuasive  group  P2  is  worse 
than  that  of  the  non-persuasive  groups.  The  reason  behind 
the  reduced  performance  of  P2  was  a  technical  problem 
involved  with  the  defibrillator  during  the  test  session.  They 
had  difficulty  in  turning  on  the  defibrillator  because  of  the 
lack  of  a  power  supply.  This  problem  was  resolved  in 
around  a  minute’s  time.  Although  the  group  performed  all 
the  remaining  tasks  on  time  after  the  issue  got  resolved,  the 
scoring  system  based  on  the  strict  time-based  protocol 
penalized  them  for  the  time  they  missed  during  the 
technical  problem. 

Road  to  the  future 

Our  study  shows  that  virtual  world  training  can  be  a  huge 
supplement  to  conventional  method  of  training.  This  is  the 
beginning  of  the  design  of  training  systems  that  integrates 
multisensory  devices  to  a  virtual,  collaborative  training 
environment  for  time  critical  procedures.  We  foresee  a  vast 
array  of  improvements  that  can  be  made  to  the  simulator.  In 
the  following,  we  describe  some  of  these: 

Proper  evaluation  metrics  for  ACLS 

At  present,  in  real  life  ACLS  training,  the  evaluation  is 
qualitative  rather  than  quantitative.  For  this  reason  it  is  very 
difficult  to  assess  the  performance  of  the  teams  with  valid 
quantitative  metrics.  The  only  metric  that  is  used  to 
evaluate  the  performance  is  the  time  taken  to  perform  each 
task  during  the  procedure.  Our  system  also  uses  the  time  as 
the  major  metric  to  evaluate  the  performance  of  the  teams. 
However,  we  believe  using  only  time  as  the  metric  is  not 
determining  valid  measurement  of  the  quality  of  team 
training.  The  level  of  interaction  and  team  dynamics  are 
also  key  factors  in  a  team-based  training  environment. 


Include  experts  on  ACLS  training  to  validate  the  simulator 
In  the  current  study,  we  enrolled  novice  participants  for  the 
validation  procedure.  Therefore,  we  saw  a  huge  difference 
between  the  performance  of  the  control  groups  and  the 
experimental  groups.  Now  that  it  has  been  observed  that 
virtual  world  training  aids  in  conventional  training  for 
novice  participants,  it  needs  to  be  further  validated  with  the 
actual  practitioners  in  the  hospitals  who  are  experts  in 
ACLS  procedure.  Although  it  is  likely  that  in  such  testing 
scenarios  of  practicing  clinicians  the  difference  in 
performance  between  the  control  groups  and  experimental 
groups  will  decrease;  nevertheless,  we  believe  that  the 
major  hypothesis  that  virtual  world  based  team  training  will 
improve  the  performance  of  a  team  will  still  be  evident. 

Integration  of  this  system  into  training  curriculum 
At  present,  the  training  is  provided  to  the  emergency  team 
practitioners  in  a  technologically  equipped  room.  Because 
of  the  cost  involved  with  the  equipment  used  in  the  training, 
it  is  less  feasible  to  provide  adequate  number  of  training 
sessions  to  all  practitioners  until  they  become  more 
confident  in  performing  the  ACLS  procedure.  The  virtual 
ACLS  training  system  has  the  potential  to  be  a  great  cost- 
effective  supplement  to  the  conventional  approach  to 
training.  The  users  can  learn  and  practice  the  ACLS 
procedure  individually  or  in  a  team.  In  addition  to  learning, 
evaluation  of  the  learned  skills  is  also  an  important  feature 
provided  by  the  system.  For  this  reason,  the  virtual  ACLS 
training  system  has  a  potential  to  be  integrated  to  the 
conventional  approach  of  training  as  a  part  of  a  training 
curriculum. 

Other  implementation  related  issues 

One  limitation  of  the  current  system  is  that  it  requires  high 
Internet  bandwidth  to  perform  collaborative  tasks  in  real 
time.  The  participants  will  experience  some  lag  in  the 
system  in  the  absence  of  high-speed  Internet  connectivity. 
The  lag  might  introduce  confusion  among  the  individuals 
during  the  team  training.  From  the  statistics  on  the  feedback 
questionnaire,  we  found  that  the  participants  experienced 
significant  time  lags  during  the  training  sessions.  A  better 
interaction  with  the  training  system  would  require 
improvements  in  the  time  lags  experienced  by  the  trainees. 

CONCLUSION 

This  work  presents  a  novel  approach  for  delivering  time 
critical  team  training  in  medical  education.  It  provides  a 
remotely  accessible  automated  centralized  platform  that  is 
capable  of  training  and  testing  teams.  It  focuses  on  various 
aspects  of  team  dynamics  such  as  time  criticality, 
communication,  procedural  task  work  and  leadership. 
Historically,  individual  training  has  been  given  preference 
over  team  training.  This  decision  is  backed  by  valid 
limitations  such  as  the  team  being  dispersed  over  several 
geographical  locations  and  the  fact  that  overlapping 
schedules  of  team  members  often  make  it  impossible  for  all 
team  members  to  be  physically  present  for  training  sessions 
at  the  same  time.  This  particular  system  overcomes  these 
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problems  since  the  training  modality  is  accessible  from 
disparate  locations.  Participants  can  log  in  remotely  at 
some  agreed  upon  time  and  participate  in  several  exercises 
as  a  team. 

ACLS  professionals  were  consulted  during  the  design  and 
implementation  phase  of  the  two  ACLS  procedures  - 
Ventricular  Fibrillation  and  PEA.  The  ACLS  professionals 
were  also  consulted  while  designing  the  scoring  scheme  and 
performance  evaluation  scheme  for  the  two  procedures.  The 
implementation  of  these  procedures  was  revised  iteratively 
until  it  was  extremely  close  to  reality.  The  ACLS 
professionals  were  very  satisfied  with  the  finished  system 
and  mentioned  the  probability  of  incorporating  the 
simulator  in  their  curriculum  for  training  the  ACLS 
procedure  to  professional  nurses  at  various  simulation 
centers. 

Similarly,  the  experimental  groups  gave  a  very  positive 
feedback  after  going  through  the  ACLS  training  in  the 
virtual  world.  The  persuasive  group  was  very  pleased  with 
the  aid  provided  by  persuasive  elements.  It  helped  them 
perform  better  in  the  initial  training  sessions.  In  the  user 
feedback  the  system  received  an  average  overall  rating  of 
8.25  on  a  scale  of  10.  Moreover,  the  participants  voted  that 
they  would  like  to  participate  in  a  team  training  sessions  as 
opposed  to  individual  training  in  the  virtual  world  as  it  is 
more  interactive  and  engaging. 

This  system  indeed  overcomes  some  of  the  previous  issues 
faced  by  similar  studies,  however  it  does  still  have  certain 
limitations  such  as  the  users’  experienced  lag  time  in 
activities  during  training  sessions.  Unless  this  issue  is 
resolved,  integrating  a  higher  number  of  multisensory 
devices  might  impact  the  training  negatively  as  the  training 
in  this  case  is  time-critical  and  significant  time-lag  in 
activities  is  unacceptable.  While  the  user  interface  can  be 
reworked  to  provide  better  usability,  it  is  a  computer  based 
simulation  and  is  not  very  easy  to  get  360  degree  view  from 
the  user’s  camera  perspective.  Moreover,  this  system  is 
incapable  of  replacing  a  professional  trainer  for  complex 
activities  such  as  ACLS.  However,  it  can  augment  the 
trainer’s  abilities  to  a  great  extent;  as  the  system  is  capable 
of  providing  training  and  testing  with  automated  evaluation 
and  feedback. 

In  the  future,  this  system  is  expected  to  be  used  to  facilitate 
ACLS  training  for  nurses  at  various  hospitals.  It  would  be 
interesting  to  see  the  results  when  this  system  will  be  used 
and  evaluated  for  professional  nurses.  It  would  also  be 
interesting  to  see  more  focus  and  variations  of  persuasive 
elements.  Additional  sensors  can  be  integrated  to  provide 
training  for  psychomotor  skills.  For  example,  we  can  now 
integrate  accelerometers  for  teaching  chemistry  lab 
experiments.  This  system  can  also  be  used  to  simulate 
several  tactical  missions  to  train  military  teams  across 
disparate  locations.  It  potentially  then  should  be  able  to  be 
used  to  train  high  performance  teams  in  various  domains 


such  as  education,  healthcare  and  emergency  services  (like 

fire  department  and  EMT). 
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